milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
22,204 Commits
169 Branches
151 Tags
614 MiB
Commit Graph

125 Commits (2.5)

Author SHA1 Message Date
XuanYang-cn 0066c016b6
enhance: Skip submit empty l0 tasks in DC (#31280) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-03-19 10:13:14 +08:00
Xiaofan 8c43c5b6cb
fix: get compaction failure when datanode is actually alive (#31353) 
didn't mark the compact as failure if it's simply an rpc error when
GetCompactionPlansResults
see #31352

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-03-18 10:01:36 -07:00
congqixia 196f0c1e1d
fix: Skip invalid compaction plan (#31045) 
See also #31044

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-06 21:36:59 +08:00
XuanYang-cn def72947c7
fix: Trigger l0 compaction when l0 views don't change (#30729) 
Trigger l0 compaction when l0 views don't change

So that leftover l0 segments would be compacted in the end.

1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager

See also: #30098, #30556

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-03-05 16:37:00 +08:00
XuanYang-cn 2867f50fcc
fix: Clear DN unkown compaction tasks (#30850) 
If DC restarted,  those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.

See also: #30137

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-03-01 11:31:00 +08:00
wayblink b74264881c
enhance: Refine compaction interfaces to support major compaction (#30632) 
Refine compaction interfaces in datacoord, support compaction result
with more than one segment. Prepare for major compaction.

related: #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-02-19 20:52:50 +08:00
XuanYang-cn fb5e09d94d
fix: call injectDone after compaction failed (#30277) 
syncMgr.Block() will lock the segment when executing compaction.

Previous implementation was unable to Unblock thoese segments when
compaction failed. If next compaction of the same segments arrives,
it'll stuck forever and block all later compation tasks.

This PR makes sure compaction executor would Unblock these segments
after a failure compaction.

Apart form that, this PR also refines some logs and clean some codes of
compaction, compactor:

1. Log segment count instead of segmentIDs to avoid logging too many
segments
2. Flush RPC returns L1 segments only, skip L0 and L2
3. CompactionType is checked in `Compaction`, no need to check again
inside compactor
4. Use ligter method to replace `getSegmentMeta`
5. Log information for L0 compaction when encounters an error

See also: #30213

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-02-01 14:25:04 +08:00
congqixia 0c7a96b48d
enhance: Make compaction log has traceID (#30338) 
See also #30167

After support open telemetry tracing, we want to have traceID as well,
this PR adds util functions to set traceID with span & propagate traceID
between different context.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-30 10:09:03 +08:00
chyezh f2985d8454
fix: compact operation on datacoord meta should preform as a transcation (#29775) 
issue: #29691

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-01-26 16:59:00 +08:00
XuanYang-cn 0b6beb7e0f
fix: Fill in info in CompactionSegmentBinlogs (#30279) 
After #28873, PartitionID and CollectionID should be filled in
CompactionSegmentBinlog so that DataNode can compose
the correct logPath. However There're some places left forgotten to fill
in the information, causing Datanode downloading `xxx/0/0/xxxx/xxxx`
binlogs during compaction

See also: #30213

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-01-26 10:51:02 +08:00
congqixia 6a73860815
enhance: Add open telemetry tracing for compaction (#30168) 
Resolves #30167

This PR add tracing for all compaction from the task start in datacoord
and execution procedures in datanode.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-23 10:37:00 +08:00
smellthemoon e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
aoiasd cb18f18c1d
fix: compacted segment status was flushing instead flushed and L0 segment trigger gc slowly (#29587) 
relate: https://github.com/milvus-io/milvus/issues/29492

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-01-09 10:52:49 +08:00
XuanYang-cn f1b6ccf305
enhance: compaction use ChannelManager interface (#29530) 
Rewrite compaction_test.go

See also: #29447

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-01-02 18:08:49 +08:00
XuanYang-cn 4b406e5973
enhance: Add CompactionTaskNum metrics (#29518) 
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-28 15:46:55 +08:00
XuanYang-cn ae180d1628
enhance: Change ChannelManager to interface (#29300) 
Rewrite cluster test
issue: #28854

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-25 19:24:46 +08:00
SimFG 67ab0e424b
fix: Clean the compaction plan info to avoid the object leak (#29365) 
issue: #29296

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 12:00:43 +08:00
XuanYang-cn 2c62bcbbf6
fix: Fix some left comments (#29027) 
See also: #28814

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-13 18:48:39 +08:00
aoiasd 3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869) 
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-11 17:52:37 +08:00
XuanYang-cn 5bac7f7897
fix: Fix L0 compaction in datacoord (#28814) 
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-05 18:44:37 +08:00
congqixia 3f9da2b8a0
fix: Check whether compaction plan exist before update (#28826) 
Fix: #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 18:54:27 +08:00
XuanYang-cn 321c5c32e3
fix: Separate schedule and check results loop (#28692) 
This PR:

- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.

- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.

  - Skips meta change when meta's already changed, avoid panic
  - Remove not inuse injectDone(bool) parameter

See also: #28628, #28209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-29 10:50:29 +08:00
XuanYang-cn 9b371067d2
feat: Add Compaction views and triggers (#27906) 
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 17:30:25 +08:00
XuanYang-cn 0be22ccacd
Fix compaction unable to notify datanode (#28409) 
See also: #28214, #28368

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 16:38:18 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
congqixia c41df18b6d
Add compaction meta in `SyncSegmentsRequest` (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
XuanYang-cn 22c089894e
Add scheduler (#27938) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-07 03:18:18 +08:00
yah01 90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
cai.zhang d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
XuanYang-cn 9598a8b236
Fix timeout task never release queue (#26593) 
See also: #26413, #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-30 11:12:27 +08:00
XuanYang-cn 35f3e263cd
Fix lock usage in DataCoord compaction (#26032) 
See also: #26009

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-01 08:55:04 +08:00
congqixia 37636b2f4d
Add compaction worker max parallel config item (#25834) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:17:00 +08:00
congqixia 8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 14:25:20 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
Xiaofan f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
MrPresent-Han b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
SimFG 4a90490a67
Fix the `segment not found` error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
aoiasd 1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:00:33 +08:00
congqixia 7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever (#22039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-07 19:02:31 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Enwei Jiao 89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
congqixia 5e49b095f5
Fix compaction handler may double release and block forever (#21015) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-06 20:09:18 +08:00
SimFG 6d9d24b4ca
Make memory and etcd data as consistent as possible (#20683) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:19:12 +08:00
Ten Thousand Leaves a8c7199a52
Fix segment_num, stored_rows_num and stored_rows_count metrics (#20704) 
TODO: better track segment with state other than flushed and dropped.

/kind bug

issue: #20146, #20145
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-22 19:21:13 +08:00
Bingyi Sun 2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
Ten Thousand Leaves 480a2b8632
Improve compaction logs (#20490) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-15 11:13:07 +08:00
Ten Thousand Leaves 5708352a80
Improve DataCoord and DataNode logs (#20265) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-03 12:15:35 +08:00
Xiaofan 11efa0bb5f
Fix compacted segment not dropped (#20128) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-27 17:15:32 +08:00
jaime d8ca56e084
Fix empty segment handoff fails (#20073) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-26 16:49:31 +08:00
Xiaofan 192151bc11
Refine Data Log (#19996) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 19:31:30 +08:00
yah01 e8ca754dca
Fix compaction produces empty segment (#19936) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-20 22:49:28 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
XuanYang-cn a0ce533727
Add a SyncSegments to sync meta between DN and DC (#19298) 
See also: #19072

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-27 16:02:53 +08:00
SimFG d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
jaime 362c663b22
Remove deprecated inner compaction type (#19024) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-07 19:16:34 +08:00
Enwei Jiao 096149e33f
fix GetCompactionState hang (#18829) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-26 14:22:55 +08:00
cai.zhang c924f73105
Refactor for IndexCoord to support cloud (#18643) 
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-25 15:48:54 +08:00
Enwei Jiao d9c7519107
refactor compaction concurency logic (#18660) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-23 15:50:52 +08:00
cai.zhang ed94ecf847
Segments can't be compacted when they have reference lock (#17649) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 21:56:12 +08:00
Xiaofan 54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
congqixia 5751759c85
Implement mix compaction logic (#15542) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-18 14:51:49 +08:00
godchen 88f4c1ce38
[skip e2e]Sort imports (#14816) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:18:08 +08:00
godchen 11b1a13835
[skip e2e]Sort imports (#14319) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-30 16:51:54 +08:00
godchen d59647bb3f
[skip ci]Fix datanode compaction comment typo (#13631) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-21 09:19:10 +08:00
godchen e36f7287ae
[skip e2e]Fix datanode compaction variable typo error (#13630) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:14:59 +08:00
edward.zeng 5db458a44e
[skip ci] Add license for data coord compaction (#12561) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-01 19:33:32 +08:00
Bingyi Sun 5574a104c9
Add triggerInfo in compactionTask (#11517) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-11 15:54:42 +08:00
Bingyi Sun c3f0c5a3ff
Add compaction interfaces in proxy (#11431) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-09 14:47:02 +08:00
sunby c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00