bdc70dfc6a
feat: Add global mmap enable configuration ( #31267 )
...
https://github.com/milvus-io/milvus/issues/31279
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components ( #30630 )
...
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
1936aa4caa
enhance: Check channel cp lag before generate compaction task ( #30997 )
...
See also #30996
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-05 13:39:01 +08:00
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
6387403639
fix: Prevent clone when selecting segments from meta ( #30928 )
...
See also #30538
Previously the `SelectSegments` changed to clone all return value
preventing possible update to returned info.
Since meta is implemented following COW rules, this shall not happen and
any update on segment shall have copy before it.
This PR:
- Remove clone for read-only Get segment info
- Add Segment Operator abstraction for changing segment
- Implemnt COW for updating MaxRowNum
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 13:59:02 +08:00
2867f50fcc
fix: Clear DN unkown compaction tasks ( #30850 )
...
If DC restarted, those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.
See also: #30137
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-01 11:31:00 +08:00
6959630652
fix: donot set l0 segment as growing when savebinlogs ( #29194 )
...
This PR fixes negative growing L0 segments in Metrics
See also: #29204 , #30441
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-04 10:21:06 +08:00
033eae9e73
enhance: Set segment.maxSize param to 1024M ( #30139 )
...
issue: #25639
/kind improvement
When the number of vector columns increases, the number of rows per
segment will decrease. In order to reduce the impact on vector indexing
performance, it is necessary to increase the segment max limit.
If a collection has multiple vector fields with memory and disk indices
on different vector fields, the size limit after segment compaction is
the minimum of segment.maxSize and segment.diskSegmentMaxSize.
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-29 10:17:02 +08:00
0b6beb7e0f
fix: Fill in info in CompactionSegmentBinlogs ( #30279 )
...
After #28873 , PartitionID and CollectionID should be filled in
CompactionSegmentBinlog so that DataNode can compose
the correct logPath. However There're some places left forgotten to fill
in the information, causing Datanode downloading `xxx/0/0/xxxx/xxxx`
binlogs during compaction
See also: #30213
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:51:02 +08:00
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
1c1f2a1371
enhance:change some logs ( #29579 )
...
related #29588
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
69a71e11d1
enhance: Add config for auto upgrade segment index ( #29112 )
...
This PR add config for auto upgrade segment index to index engine's
version.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 11:56:37 +08:00
ef18e6a532
enhance: Use a non-blocking method to trigger compaction when saveBinlogPath is executed ( #28941 )
...
/kind improvement
issue: #28924
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-12 10:16:39 +08:00
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
adce6aab4f
fix:handle err when ManualCompaction ( #28804 )
...
handle err when ManualCompaction
#28644
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-30 17:06:31 +08:00
9b371067d2
feat: Add Compaction views and triggers ( #27906 )
...
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
7445d3711c
feat: trigger compaction to handle index version ( #28442 )
...
issue: https://github.com/milvus-io/milvus/issues/28441
---------
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-21 09:26:22 +08:00
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
b2e7cbdf4b
Remove TimeTravel in compactor ( #26785 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact ( #26203 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-08 21:17:21 +08:00
6f18587f35
Fix small segment compaction ( #21327 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-26 14:49:01 +08:00
b15e34db21
Add contraint for compaction based indexed segments ( #25709 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-23 21:31:00 +08:00
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
7e645dabf1
Add trace log for datacoord ( #24046 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-05-12 09:53:21 +08:00
5aa9db0d38
Add collection level auto compaction enabled config ( #24013 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-10 17:45:20 +08:00
72485c9eeb
Remove index check to trigger compaction ( #23657 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-04-24 20:26:34 +08:00
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
319f1773af
Fix segment not found if forward delete to empty segment ( #22528 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
e5f408dceb
Merge IndexCoord and DataCoord ( #21267 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
ff2a68e65a
Fix collection not exist when tried to do recovery ( #21471 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
ef3fcdf65b
Small candidate compaction should only happen with >1 segments ( #21249 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-16 10:01:23 +08:00
a81c9cc11c
Recalculate segment max size during compaction ( #21077 )
...
issue: #21077
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-13 15:39:21 +08:00
e977e014a9
Fix flush didn't respect binaryvector and other schemas ( #21120 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-12-12 10:33:26 +08:00
d67db2ed9b
Make compaction target able to exceed segment max size by a bit ( #21068 )
...
issue: #21053
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-09 16:03:20 +08:00
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
2f5b03fef7
Split small segment proportion and small segment compactable proportion ( #20996 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-06 14:37:18 +08:00
d1cb380ee0
Enable to gen compaction plan for stale segments ( #20741 )
...
See also: #20502
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-22 17:33:16 +08:00
480a2b8632
Improve compaction logs ( #20490 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-15 11:13:07 +08:00
5f5fdb0789
Compaction bypass stale segments ( #20202 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-31 19:13:34 +08:00
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
2a2d32c4fc
Fix datacoord panics when collection info not found ( #19706 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-11 21:39:24 +08:00
31db01b6ae
Support TTL for collection level ( #19383 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
59f8e96522
Change compactor policy from loading all binlogs to loading serially ( #13716 ) ( #19280 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-09-25 20:12:52 +08:00
ee9d634a43
Compaction requires the source segments indexed ( #19368 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-22 18:34:50 +08:00
71d4a32ce6
Fix deadlock ( #19340 )
...
The IsParentDropped() acquires read lock of meta,
which leads to recursively acquire read lock
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-22 15:48:50 +08:00
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
11b352c3ae
Guarantee flushed segments are all indexed ( #19062 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-16 11:32:48 +08:00
096149e33f
fix GetCompactionState hang ( #18829 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-26 14:22:55 +08:00
Bingyi Sun
Buqian Zheng
congqixia
jaime
XuanYang-cn
xige-16
smellthemoon
wei liu
SimFG
Enwei Jiao
wayblink
cai.zhang
yah01
Jiquan Long
Ten Thousand Leaves
Xiaofan
zhagnlu