milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,028 Commits
133 Branches
115 Tags
559 MiB
Commit Graph

736 Commits (a2f0092e39d70efcef57b5e1e0dddd032cf7bcc7)

Author SHA1 Message Date
wayblink 2bc212c6c9
enhance: Add L2 segment level (#29595) 
#28410

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-02-18 14:12:49 +08:00
xige-16 060c8603a3
fix: Support mvcc with hybrid serach (#30114) 
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-02-01 16:03:03 +08:00
yihao.dai c5918290e6
feat: Add import executor and manager for datanode (#29438) 
This PR introduces novel importv2 roles for datanode:
1. Executor: To execute tasks, a import task will be divided into the
following steps: read data -> hash data -> sync data;
2. Manager: To manage all the tasks;

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-31 20:45:04 +08:00
cai.zhang f619d792c0
enhance: Break down the granularity of collection info cache expired (#29977) 
issue: #29772 

1. `DropPartition` only invalidates the cache related to the partition.
2. `CreateAlias` does not invalidate the cache.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-01-30 16:45:02 +08:00
Patrick Weizhi Xu 0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931) 
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-01-24 00:04:55 +08:00
yihao.dai 8780d65b66
fix: Use channel cp as the dml&start position for import segments (#30107) 
This PR discontinuing the subscription to the mq and, instead, employing
the channel checkpoint as the DML and starting position for the import
segments.

issue: https://github.com/milvus-io/milvus/issues/30106

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-22 14:36:55 +08:00
XuanYang-cn 86f48861c1
fix: Add more throughput in related metrics (#30038) 
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode

See also: #30099

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-01-19 11:34:54 +08:00
smellthemoon e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
Bingyi Sun e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
wayblink 1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641) 
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-11 19:12:51 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
xige-16 9702cef2b5
feat: Support multiple vector search (#29433) 
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-08 15:34:48 +08:00
MrPresent-Han 9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
congqixia da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626) 
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 17:02:46 +08:00
congqixia aa967de0a8
enhance: Explicitly pass LevelZero segment ids in vchan info (#29612) 
See also #27675

For `GetRecoveryInfo` & `GetRecoveryInfoV2`, Level zero segment ids
shall be specified in vchan info so that querycoord could re-fetch
current segment info during watch procedure without having all segment
info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 16:46:45 +08:00
xige-16 02673914a0
feat: Support multiple vector indexes in a collection (#27700) 
issue: #25639 

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-12-29 11:44:45 +08:00
congqixia 02bc0d0dd5
fix: Add scope limit for querynode DeleteRequest (#29474) 
See also #27515

When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 14:28:47 +08:00
yah01 a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
congqixia 8a63e53421
enhance: Add http method to control datacoord garbage collection (#29052) 
See also #29051

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 19:26:39 +08:00
Bingyi Sun ad866d2889
feat: integrate storagev2 into index build process (#28995) 
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-13 17:24:38 +08:00
Bingyi Sun 36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
yihao.dai d26b563a8b
feat: Define import API and metadata (#28731) 
Define the new rpc and metadata for ImportV2.

see also: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-04 19:56:35 +08:00
aoiasd b4af6f8c40
fix: sync action load segment with lack collection index info list (#28788) 
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-04 18:14:34 +08:00
Bingyi Sun 45e6801ce4
feat: Add checker activation service interfaces (#28850) 
issue: #28610

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-04 17:38:37 +08:00
wei liu 911a915798
feat: enable balance based on growing segment row count (#28623) 
issue: #28622 

query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-27 14:58:26 +08:00
XuanYang-cn 24616e7838
fix: Remove ignored files in path (#28696) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-24 15:04:26 +08:00
XuanYang-cn 9b371067d2
feat: Add Compaction views and triggers (#27906) 
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 17:30:25 +08:00
Bingyi Sun 4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
Bingyi Sun e8c5e75786
enhance: ignore proto generated files (#28537) 
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-22 10:20:22 +08:00
yah01 bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
congqixia c41df18b6d
Add compaction meta in `SyncSegmentsRequest` (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
yah01 ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
aoiasd 1d4be0d257
Adjust datacoord for L0 Delta (#28021) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-06 15:26:16 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
zhagnlu 6060dd7ea8
Add chunk manager request timeout (#27692) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-23 20:08:08 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
congqixia b7bfccaf21
Add channel name paramter in `FlushSegmentsRequest` (#27725) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-17 12:00:10 +08:00
jaime e386a62fae
Remove recollect segment stats during starting datacoord (#27410) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:26:09 +08:00
XuanYang-cn 294ff74ca5
Add new compaction types (#27608) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-11 06:49:32 +08:00
congqixia cfe475a2ea
Add `SegmentLevel` field for L0 delta (#27502) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:55:31 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
Jiquan Long 370fdaf50d
Record engine version for segment index (#27384) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-28 18:03:28 +08:00
Enwei Jiao b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
wei liu 4071132f6a
reload loading collection when qc recover (#27300) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-27 11:43:28 +08:00
congqixia fd979a725a
Fix query_coord.proto comment format caused lint issue (#27375) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-26 20:35:27 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
cai.zhang a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
MrPresent-Han 7939f0e7d5
enable ctx traceId for assignsegment on dc(#26972) (#27108) (#27030) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-18 11:39:20 +08:00
Enwei Jiao 0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
Jiquan Long af5619e73e
Fix misusing chunk manager (#27051) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-13 18:11:20 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
Enwei Jiao a73eb1c831
Sync pb files (#26643) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-29 12:36:26 +08:00
congqixia 8d13717cac
Fill Collection start position timestamp in WatchInfo (#26370) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 09:05:32 +08:00
cai.zhang a0198ce8ae
Support json contains feature (#25384) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-11 17:09:30 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
Jiquan Long 6578a27fe7
Support empty expression (#26153) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-10 08:37:15 +08:00
zhenshan.cao 7619db1edc
Fast GC compacted dropped segment (#26207) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-08-09 19:39:15 +08:00
Jiquan Long 5c1f79dc54
Push down the limit operator to segcore (#25959) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-01 20:29:05 +08:00
cai.zhang 760a2d9aa7
Support AllocTimestamp api for Milvus (#25784) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-25 10:05:00 +08:00
wei liu fc19b85a40
fix count(*）retrieve redundant growing segment (#25825) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-24 14:09:00 +08:00
congqixia cc71e065e3
Support load index after segment loaded (#25567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 10:51:19 +08:00
MrPresent-Han 49655d2f13
fix panic due to lack of indexInfo on querynode(#25362) (#25613) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-17 19:23:19 +08:00
wei liu b62c82af22
fix set target version in loading sealed segment (#25603) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-14 20:00:31 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
congqixia 5aec6036dc
Add index info in GetDataDistribution response (#25444) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-11 11:22:29 +08:00
smellthemoon d63323d117
Add rate limit and deny write in upsert (#25351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-07-11 11:20:34 +08:00
chyezh d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-03 18:24:25 +08:00
wei liu 68ae199a9f
load segment with target version, avoid read redundant segment (#24929) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 11:48:45 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan 72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
wei liu e40d95e093
add search/query request cost metrics on qn (#24413) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-15 14:24:38 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
cai.zhang a6dbcaeb7a
Add proto field to support reverse in (#24678) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-06-07 17:24:36 +08:00
xige-16 732fe54775
Support partition Key (#24047) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-06 10:24:34 +08:00
cai.zhang 93ea9c4925
Support return pending index rows when describe index (#24588) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-06-01 18:14:32 +08:00
yihao.dai b0ce4cff83
Assign metric type in search request when user has not explicitly assigned (#24591) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-01 18:10:32 +08:00
yihao.dai e49df11942
Fix QueryNode is not able to recovery (#24335) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-24 14:29:29 +08:00
yah01 abd250da09
Fix QueryNode is not able to recovery (#24300) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-23 17:17:26 +08:00
wei liu ce03248b1a
simplify logic circle in search (#24110) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-23 16:01:26 +08:00
Jiquan Long bd343550a5
Support to manage connections (#24224) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-05-19 12:51:23 +08:00
yihao.dai 1a3dca9b5e
Fix dynamic partitions loading (#24112) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-18 09:17:23 +08:00
MrPresent-Han b74770b636
support try-best query reducer methods(#24075) (#24103) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-05-17 16:23:23 +08:00
foxspy 849de66be8
revert query_coord proto (#24089) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-05-15 15:09:25 +08:00
cai.zhang 9715a850fa
Support expr with json field (#23804) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-05-10 10:19:19 +08:00
yihao.dai 7e0d1492c7
Load delete from channel checkpoint (#23961) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-09 19:10:41 +08:00
wayblink 899702f13c
Implement GetIndexStatistics interface (#23603) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-06 10:34:39 +08:00
yihao.dai 4b26c0afb3
Add collection in querynode when sync new partitions (#23825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-01 16:36:38 +08:00
foxspy 6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
wei liu 4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
yah01 e23af6406f
Add nested path for accessing json fields (#23651) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-24 17:14:34 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
yah01 296380d6e6
Support async refresh (#23107) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-12 15:06:28 +08:00
MrPresent-Han ba02e70301
fix bug when syncing distribution without schema and remove unused logic(#23085) (#23216) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-04-07 19:32:29 +08:00
yihao.dai a182594bf5
Add FlushAll support (#22942) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-30 11:28:25 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Gao 8b3e5189e1
Add querynode plugin (#22379) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-03-24 15:21:59 +08:00
yihao.dai 1f718118e9
Dynamic load/release partitions (#22655) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-20 14:55:57 +08:00
Jiquan Long 8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
aoiasd cc5d72a9f2
add search option to ignore ignore segment (#21994) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-22 17:31:46 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
Xiaofan f1daef22a1
Fix failed bulkload segment marked as sealed (#21799) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-31 12:41:53 +08:00
wei liu 73c44d4b29
resource group impl (#21609) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-01-30 10:19:48 +08:00
jaime 1dcd9eeb79
Support rename collection (#21693) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-19 14:13:43 +08:00
Ten Thousand Leaves defb7660c8
Add refresh option for LoadCollection/LoadPartitions interfaces (#21776) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-01-18 16:41:44 +08:00
Jiquan Long 9fd9bed2b9
Remove collection meta after GC finished (#21595) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-12 09:55:42 +08:00
cai.zhang e127cf7b99
Reset indexpb for upgrade (#21620) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-11 14:35:40 +08:00
bigsheeper 79535931ab
Return error code when deny to read/write (#21561) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-01-06 14:31:37 +08:00
cai.zhang aa203acfb3
Low IndexCoord weight (#21548) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-06 14:21:37 +08:00
cai.zhang e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Jiquan Long ff2a68e65a
Fix collection not exist when tried to do recovery (#21471) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-04 16:37:35 +08:00
bigsheeper c187de7754
Show trigger reason when deny to read/write (#21368) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-30 18:35:32 +08:00
bigsheeper fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete (#21110) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-12-13 16:17:22 +08:00
Ten Thousand Leaves d67db2ed9b
Make compaction target able to exceed segment max size by a bit (#21068) 
issue: #21053
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-09 16:03:20 +08:00
jaime 0a5270e8c0
Fix alter collection hang (#20694) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-11-21 10:09:10 +08:00
yah01 2d806788fb
Add version for subscribed channel (#20585) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-15 13:21:07 +08:00
cai.zhang 2b02b869ec
Return real-time indexed rows and total rows (#20460) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-11-14 14:37:07 +08:00
bigsheeper cd19d99ad7
Add channel level checkpoint (#20350) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-10 22:13:04 +08:00
groot 53850e340c
Dont expire pending import tasks (#20431) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-11-10 10:25:08 +08:00
wei liu c5cd92d36e
update target (#19296) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-11-07 19:37:04 +08:00
cai.zhang bcffb3c389
Check indexes in indexcoord instead of proxy (#20193) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-10-31 11:39:33 +08:00
groot bee66631e3
Refine bulkload (#19671) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-10-27 16:21:34 +08:00
jaime d8ca56e084
Fix empty segment handoff fails (#20073) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-26 16:49:31 +08:00
XuanYang-cn e081eb287d
Refine flowgraph nodes for readability (#19864) 
- delete compaction tasks from executor while dropping

See also: #19780

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-10-20 16:39:29 +08:00
wayblink dbd60757c9
Support pass start_ts and end_ts through Import API (#19810) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-10-20 12:15:27 +08:00
yah01 470e8ef419
Store index ID to filter out invalid handoff event (#19703) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-20 11:29:27 +08:00
jaime b79687687d
Replace segment index file path with index file id within segment index meta (#19866) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-19 16:55:27 +08:00
XuanYang-cn 2785f98de6
Refactor segment replica in DN (#19815) 
- Replace isNew and isFlushed with sType
- Add compacted into proto
- Rename replica to channel
- Rename segmentReplica to channelMeta
- Replace 4 maps with 1 map
- Add segment.go and segment_test.go

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-10-18 15:33:26 +08:00
jaime 59bcbf0cf6
Support check health for the service level (#19779) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-18 13:39:26 +08:00
cai.zhang 9d43947f1c
Must create index before load (#19516) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-10-17 18:01:25 +08:00
SimFG a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 17:51:24 +08:00
jaime 8e7a4733ec
Add GetVersion API for the client (#19694) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-14 09:59:24 +08:00
jaime 10c03de628
Refine method name from BroadCastXx to BroadcastXx (#19700) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-11 21:07:23 +08:00
SimFG 125f827ca4
Use underscores to separate parameter names in the `GetLoadingProgressRequest` (#19692) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-11 11:55:22 +08:00
jaime 31db01b6ae
Support TTL for collection level (#19383) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-10 20:31:22 +08:00
wayblink 85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-10-10 15:55:22 +08:00
congqixia d64f318a80
Add collection schema in watch channel info (#19663) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-10 01:37:21 +08:00
congqixia 8c1c8000ac
Add TargetID in MsgBase and querynode v2 API check (#19627) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-08 20:26:57 +08:00
zhenshan.cao 0ac4bc32a5
Add logic of autoindex (#19570) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-10-08 15:38:58 +08:00
wayblink e7c2500c71
Implement GetPersistentSegmentInfo through datacoord (#19205) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-10-08 11:51:02 +08:00