milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,060 Commits
146 Branches
120 Tags
575 MiB
Commit Graph

50 Commits (cec7e99106b2d5f477b299a74255313b237bf898)

Author SHA1 Message Date
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
yah01 0a750408d0
Skip delta logs have been applied (#26971) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 16:21:23 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 11:39:20 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
congqixia 394c898b4c
Discard SyncDistribution set action from legacy querycoord (#27027) 
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 09:35:18 +08:00
congqixia ac45af585b
Make segment loaded successful put in manager even ctx done (#26992) 
Leave segment loaded in manager even wait other segment failed
See also #26908
Fix error case in distributed scenario

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 21:33:18 +08:00
congqixia 2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
congqixia c6116d1819
Remove segment to LocalSegment type assertion (#26931) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-09 10:35:16 +08:00
congqixia c8f9f22c4a
Fix segment loader return false success (#26926) 
`waitSegmentLoadDone` did not check waitCh result is success or failure
after load return without error, delegator will assume all segments are loaded

This PR changes waitCh to loadResult with `sync.Cond` with `atomic.Int32` to represent status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-08 16:41:16 +08:00
MrPresent-Han 528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
XuanYang-cn ef75784715
Fix LoadSegmentLatency metric p99 (#26761) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-01 11:07:07 +08:00
MrPresent-Han 8330c18dc9
add log for loading segment(#26564) (#26640) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-31 12:03:00 +08:00
yah01 bfcc691129
Fix segment leaked if task canceled (#26685) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 14:17:03 +08:00
congqixia 1cf6e00fa6
Improve segment manager interface (#26637) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 15:46:27 +08:00
congqixia ec65a4e048
Use float64 for resource estimation log (#26335) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 14:25:33 +08:00
Enwei Jiao 7d61355ab0
Refactor log for Query (#26310) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-14 18:57:32 +08:00
yah01 a173486d2e
Fix calculation of memory usage prediction for mmap mode (#26264) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-12 17:19:31 +08:00
yah01 48422dd4c5
Fix spawn too many threads (#26293) 
- Low the thread pool cap
- Limit CGO calls concurrency

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-11 18:29:29 +08:00
wei liu b47a72bfcf
fix set dirty segment distribution to leader view (#26180) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-11 11:21:32 +08:00
xige-16 1971d98897
Add disk metric info (#25675) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-11 10:35:42 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
yah01 300fef446b
Enable mmap for vector index (#25877) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-10 13:59:15 +08:00
yah01 889424b3f9
Fix load index with empty file list (#26236) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-09 18:39:16 +08:00
MrPresent-Han 68ecf49ed5
fix lost parameter for threadCoefficient(#25781) (#26109) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 12:05:06 +08:00
yah01 9c55a7f422
Add worker num as one of load resource (#26045) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-01 21:47:06 +08:00
yah01 39b00b97a6
Add more logs for committed resources (#26026) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 19:27:03 +08:00
yah01 8245e078c0
Add max segment size log (#26015) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 14:37:04 +08:00
congqixia 5fbe6e99fd
Remove debug log from segment loader (#25937) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-26 18:51:01 +08:00
yah01 3735a097e6
Refine the memeory usage prediction (#25889) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-25 17:41:01 +08:00
congqixia 0bc03ede0d
Add eventlog pkg and support grpc streaming event observation (#25812) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-25 17:23:01 +08:00
yah01 168f346620
Remove deprecated interface (#25864) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-07-24 17:47:00 +08:00
congqixia cc71e065e3
Support load index after segment loaded (#25567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 10:51:19 +08:00
congqixia c490b30db7
Fix segment loader skip wait if all segments are loading (#25643) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-17 19:51:18 +08:00
yah01 205a7c430a
Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
yah01 dd5f896dc8
Load batch by batch (#25212) 
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 13:58:27 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
congqixia 31880ab427
Use conc.PoolOption instead of ants.Option (#24585) 
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-01 15:52:31 +08:00
congqixia 48dff300db
Fix Get segment returns ambiguous result (#24505) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-30 12:25:28 +08:00
aoiasd c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
yah01 400364483d
Improve predict the resource usage (#23968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-11 15:33:24 +08:00
yihao.dai 7e0d1492c7
Load delete from channel checkpoint (#23961) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-09 19:10:41 +08:00
congqixia 884f7904a1
Re-calculate segment entry num when binlog field is zero (#23883) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-06 14:24:39 +08:00
yihao.dai 02734d9eb0
Fix ignoring error when loading segment (#23830) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-04 15:10:39 +08:00
congqixia 4fe363c4b2
Add KNN cgo pool (#23526) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 23:56:31 +08:00
yah01 dc0eb433ae
Add IoPoolSize config (#23360) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-13 14:50:28 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00