b172ea1093
fix: Remove enableLevelZeroSegment config ( #36535 )
...
See also: #36504
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-17 11:59:24 +08:00
f3b6792a25
enhance: Enhance segment log ( #36848 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-15 20:43:30 +08:00
5ec4163d0f
feat: support bm25 logs mixcompaction ( #36072 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-14 16:57:22 +08:00
d230b91bd1
enhance: Add PreallocatedSegmentIDs for the compaction task ( #36734 )
...
Add `PreallocatedSegmentIDs` field to the compaction task, allowing the
`ResultSegments` in the compaction task to represent the final segments
produced by the compaction.
issue: https://github.com/milvus-io/milvus/issues/36733
also related: https://github.com/milvus-io/milvus/issues/36686
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-13 17:59:21 +08:00
db34572c56
feat: support load and query with bm25 metric ( #36071 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
f7b811450d
feat: add enable_tokenizer params to VarChar field ( #36480 )
...
issue: #35922
add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00
a5f98e4cb3
enhance: UpdateSegmentsInfo should update remaining segment info even if some one not exist ( #36726 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-10 15:17:19 +08:00
62679efa0c
enhance: remove unnecessary clone in meta cache ( #36628 )
...
See #36627
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-10-10 15:05:20 +08:00
5fc731795b
enhance: Datacoord to support prioritization of compaction tasks ( #36547 )
...
See #36550
This PR made 2 changes:
1. Introducing a prioritization mechanism, if
`dataCoord.compaction.taskPrioritizer` is set to `level`, compaction
tasks are always executed as the priority of L0>Mix>Clustering
2. `dataCoord.compaction.maxParallelTaskNum` now controls the
parallelism of executing tasks, not the task number of queue +
executing.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-10-09 19:11:20 +08:00
130a923dec
enhance: the estimate method when loading the collection ( #36307 )
...
- issue: #36530
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-09 17:35:19 +08:00
0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric ( #36601 )
...
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.
issue: https://github.com/milvus-io/milvus/issues/36600 ,
https://github.com/milvus-io/milvus/issues/36518
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 14:41:20 +08:00
90a0e90732
enhance: remove warnings when there is no partition key set ( #36674 )
...
issue: #36673
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-10-09 10:27:20 +08:00
ef1832ff9c
enhance: enable manual compaction for collections without indexes ( #36577 )
...
issue: #36576
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 19:57:18 +08:00
1fded42277
enhance: decrease CPU overhead when calculating index file size ( #36579 )
...
issue: #36578
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 14:29:25 +08:00
4e0ea39235
fix: Remove neighbors if compactTo is unindexed ( #36503 )
...
See also: #36360
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-08 14:15:19 +08:00
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions ( #36624 )
...
issue: #36623
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-08 11:23:23 +08:00
1f47d5510b
fix: Fix import segments leak in segment manager ( #36602 )
...
Directly add import segments from the meta, eliminating the dependency
on the segment manager.
issue: https://github.com/milvus-io/milvus/issues/34648
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-08 10:11:22 +08:00
00a5025949
enhance: support clustering compaction on null value ( #36372 )
...
issue: #36055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) ( #36214 )
...
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.
Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:
1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.
To address these limitations, This enhancement is needed.
Related Issue: #36212
2024-09-30 13:23:32 +08:00
ecb2b242e2
enhance: Add sorted for segment info ( #36469 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
acc9b5ada5
fix: Regenerate partition stats after stats task for clustering compaction ( #36408 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:29:21 +08:00
7bf40694fd
fix: Record the nodeID before assigning tasks ( #36371 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:21:15 +08:00
7ff41697f9
enhance: make slotUsage a field of CompactionTask ( #36510 )
...
#36509
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
9e8cafcbe2
enhance: Skip loading bf in datanode ( #36367 )
...
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
d2c774fb6d
fix: Return all compactTo segments after support split ( #36361 )
...
Related to #36360
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-20 14:11:11 +08:00
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
a61668c77e
feat: Introduce stats task for import ( #35868 )
...
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed
issue: https://github.com/milvus-io/milvus/issues/33744
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 15:17:08 +08:00
9d8d332c88
fix: Fix improper use of offset in HybridSearch ( #36244 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:15 +08:00
22cce44afc
fix: metrics stored_index_files_size is never cleared ( #36160 )
...
issue: #36159
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-13 20:09:15 +08:00
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
e8840a1b41
enhance: Add metrics for Delete entries num of L0seg ( #36175 )
...
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*
See also: #36147
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-12 18:07:08 +08:00
208c8a2328
fix:support config index offsetcache and fix create same index again ( #35985 )
...
#35971
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-08 18:23:05 +08:00
6b4ae0c65e
enhance: Log warn on delayed compaction tasks ( #36049 )
...
/kind enhancement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-08 17:23:05 +08:00
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints ( #35707 )
...
issue: #35706
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 10:11:05 +08:00
f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 ( #35694 )
...
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701 )
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-27 10:36:59 +08:00
82743c5c50
fix: Clear channelcp meta and metrics ASAP ( #35658 )
...
See also: #35588
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 12:22:57 +08:00
4236c3bafb
enhance: enable MV on binary vectors ( #35668 )
...
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-25 16:27:09 +08:00
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data ( #35498 )
...
#35458
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
227ecd3901
enhance: Remove the check for channel cp lag when generating compaction plan ( #35383 )
...
issue: https://github.com/milvus-io/milvus/issues/35382
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-19 19:40:55 +08:00
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
c42976ee6f
enhance: Init ChannelCP when creating a channel ( #35387 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-14 10:16:58 +08:00
f5b4a972a0
fix: checkResultTicker leak ( #35377 )
...
https://github.com/milvus-io/milvus/issues/35376
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-13 21:36:52 +08:00
1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo ( #35369 )
...
issue: #35368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-13 17:20:19 +08:00
06f9ba294e
enhance: Add more ut for l0CompactionTask ( #35100 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 14:24:18 +08:00
61a426bc6f
enhance: Init channel cp metric when DataCoord recover ( #35353 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 10:08:18 +08:00
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
838f06323f
fix: datacoord binlog size metrics include dropped segment ( #35338 )
...
issue: #35337
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 16:24:16 +08:00
c6253f9c9b
enhance: improve ut cov of clustering compaction task ( #35242 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:44:21 +08:00
bb15ecdc13
enhance: improve ut for compaction_policy_clustering ( #35205 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:42:17 +08:00
678018d9ca
enhance: Avoid unnecessary compaction ( #35148 )
...
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.
issue: https://github.com/milvus-io/milvus/issues/35147
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 10:30:21 +08:00
a2b517523d
enhance: Add in-memory cache for casbin enforcer result ( #35271 )
...
See also #35270
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 18:48:15 +08:00
b71e058bc5
enhance: Add import option to skip disk quota check ( #35274 )
...
Add an option to skip the disk quota check for backup-restore import.
issue: https://github.com/milvus-io/milvus/issues/33775
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 16:40:16 +08:00
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
9871966415
enhance: segment alloc interceptor ( #34996 )
...
#33285
- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-04 07:40:15 +08:00
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
81773bfadf
enhance: add commit time in partitionStats proto ( #35125 )
...
fix : #35110
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 16:16:14 +08:00
f466129924
enhance: add segment state on data coord bin log size metrics ( #35153 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
5bbb1c201c
enhance:support l2 single compaction ( #34935 )
...
#34928
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
089fb2b5fb
enhance: Skip updating the level that are already at the specified level ( #35094 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:18:12 +08:00
acaa78db58
fix: disable sparse and binary in MV ( #35126 )
...
issue: #35120
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 0d340ebe7412cd85963825bc8abd3c874e339679)
2024-07-31 15:02:01 +08:00
f919b6e1b8
enhance: refine error handle in clustering compaction task ( #34956 )
...
related: #30633 #34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:37:57 +08:00
ce3f836876
fix: compaction task not be cleaned correctly ( #34765 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:21:56 +08:00
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
a495754034
enhance: the datacoord gc should fast quitable ( #35050 )
...
issue: #35049
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-29 17:03:48 +08:00
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
9a07790215
enhance: Add ut for l0CompactionTask processExecuting ( #34800 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-29 10:53:48 +08:00
497afcb897
enhance: Refine code for GetRecoveryInfo ( #34973 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-29 00:01:46 +08:00
6e9fbd1630
enhance: Solve channel unbalance on datanode ( #34984 )
...
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.
This PR refine the balance policy to solve channel unbalance on datanode
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 17:35:47 +08:00
1cff55381d
enhance: add manual alloc segment rpc for datacoord ( #35002 )
...
issue: #33285
- segment allocation will move to streamingnode, so a manual alloc
segment rpc is required
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-26 10:15:46 +08:00
d93c51d5f6
enhance: change GetVChannels rpc into GetPChannelInfo ( #34940 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 13:29:45 +08:00
bef06e5acf
enhance: add streaming coord and node grpc service register ( #34655 )
...
issue: #33285
- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 12:19:44 +08:00
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
3cd24f7548
fix: collection meta is not removed after gc in DataCoord ( #34883 )
...
issue: #34847
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
52fa668f5d
fix: fix collection is compacting logic ( #34855 )
...
fix : #34861
also related to: #34792
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:35:41 +08:00
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
c2b8b5fe84
enhance: refine clustering compaction configs and logs ( #34784 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:23:40 +08:00
b22e549844
enhance: Rename config of sealing by growing segmetns size ( #34787 )
...
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 20:27:41 +08:00
a11714913b
fix: fix unstable ut of compaction pickSlot ( #34828 )
...
#34827
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:47 +08:00
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
90e765db87
fix: fix bug if datacoord restart when l0 compaction saveSegmentMeta ( #34679 )
...
#34681
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 13:03:41 +08:00
fd7221f4be
fix: Fix accidentlly exit MixCompaction task loop ( #34688 )
...
See also: #33431 , #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 15:57:42 +08:00
b5b5afb63f
fix: [skip e2e]golang lint check fail ( #34682 )
...
#34680
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-16 10:47:36 +08:00
1495b7a2c9
fix: Fix no L0 compaction tasks are executed ( #34537 )
...
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.
issue: https://github.com/milvus-io/milvus/issues/34460
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-15 17:01:37 +08:00
8b5754f7fe
enhance: Add segment seal proportion jitter ( #34636 )
...
See also #34574
Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.
This PR also fix license header in paramtable pkg.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:47:39 +08:00
XuanYang-cn
yihao.dai
aoiasd
Buqian Zheng
Ted Xu
SimFG
Patrick Weizhi Xu
jaime
wayblink
Rijin-N
cai.zhang
congqixia
zhenshan.cao
zhagnlu
Chun Han
Zhen Ye
wei liu
CharlesFeng
smellthemoon