be688d894a
Forbid multiple indexes on same field ( #16875 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-05-10 21:07:53 +08:00
4ee78eb5ad
proxy support tls ( #16706 )
...
Signed-off-by: xiyichan <2863768433@qq.com>
2022-05-10 20:05:53 +08:00
5c98329f7c
Return error when no replica available ( #16886 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-10 19:47:53 +08:00
000c5ff3de
Fix msgstream unsubscription ( #16883 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-10 19:43:52 +08:00
34833680da
Add lock before query release collection from replica ( #16880 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-10 18:09:53 +08:00
216c45fbd6
Add tests where segments load delta logs ( #16825 )
...
issue: #16821
/kind improvements
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-10 17:59:52 +08:00
2d0f908dba
Fix updating segments' NodeIds correctly after LoadBalance ( #16854 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-05-10 15:47:53 +08:00
72bbe40254
Make PayloadReader read column data in batch ( #16826 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-10 11:37:52 +08:00
5afdf3d49f
Move locks back to meta table methods ( #16848 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-09 20:47:52 +08:00
76bfd41952
Refine DataNode logs. ( #16849 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-05-09 16:25:52 +08:00
62658dcda6
Fix pulsar unscubsribe fail because of consumer not found ( #16839 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-09 12:07:52 +08:00
b0c9f25352
Sort out the log of IndexCoord ( #16810 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-05-09 12:05:52 +08:00
8fcf349609
Fix create index failed after drop index ( #16836 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-09 11:35:53 +08:00
c7935d1a86
Add legacySegments for load balance segments rc track ( #16831 )
...
- Add legacySegments in ShardCluster tracking old segment inUse
- Apply node id check for segment online/inUse check
- Add unit test simulating load balance case
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-07 18:25:52 +08:00
ac389f3e93
Limit and translate pattern match operations ( #16818 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-05-07 16:31:52 +08:00
6b2d3c3ec1
Make each dataNode only accept one bulk load task simultaneously ( #16820 )
...
issue: #16674
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-07 14:05:52 +08:00
92b6293be4
Fix QueryNode log level ( #16604 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-07 10:27:51 +08:00
67ce206cf0
make SegmentMaxLifetime configurable ( #16783 )
...
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-06 23:41:51 +08:00
31ddff2056
Some minor fixes and improvements ( #16814 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-06 21:35:51 +08:00
64d2793fe9
Fix wrong result topks when reduce search result ( #16815 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-06 21:17:51 +08:00
a8e1c8fa9d
Make duplicate flush recalls success and other logic update ( #16755 )
...
Latest logic:
1) Duplicate flush calls on same segments will not result in errors (same as the original design)
2) `FlushSegments` now still flushes stale segments even if non-stale segments failed to get flushed
issue: #16749
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-06 17:49:51 +08:00
98ceb162aa
Support antlr as plan parser ( #16696 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-05-06 17:43:51 +08:00
b0053b7a45
Add isShardLeader field in search/query request ( #16798 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-06 16:55:51 +08:00
009b773b6e
Delete set default type param max_len for varchar field ( #16794 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-06 15:59:51 +08:00
8736372fd2
Fix bulkload bugs ( #16760 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-06 11:21:50 +08:00
7cc995d085
Prohibit creating diff indexes with same index_name on a collection ( #16774 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-05 21:33:51 +08:00
1acd256481
Add DataQueryable and DataIndexed states for bulk load tasks ( #16725 )
...
issue: #16607
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-05 21:17:50 +08:00
c82e2453eb
Modify the replicas' shard info after load balance ( #16785 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-05-05 21:15:50 +08:00
8c004fa981
Fix replicaID is allow zero from watch channel task ( #16788 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-05 18:11:51 +08:00
20936f78da
Fix LoadBalance failed to sync segments to shard leader when the leader is offline ( #16770 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-05-05 16:25:50 +08:00
333b439a6c
Support config grpc log level ( #16618 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-05 10:35:50 +08:00
fa97f86cda
Clean up not used id_cache related code ( #16655 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-05-05 09:59:51 +08:00
337ad53613
Support config small index nprobe and nlist ( #16731 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-05 09:49:50 +08:00
2be46a01c0
Reduce the number of retries and add error log ( #16754 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-05-05 09:31:51 +08:00
c25b337c36
Remove AddTenant from meta table ( #16717 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-05-05 09:23:51 +08:00
5375161777
test: use `T.TempDir` to create temporary test directory ( #16758 )
...
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Fixes : #16759
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-05-04 11:35:50 +08:00
21a1311f66
Merge utils/Utils.h into common/Utils.h ( #16762 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-05-03 12:05:50 +08:00
6a62ff18bf
Support easylogging config for segcore and knowhere ( #16751 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-05-03 08:39:49 +08:00
d5db4ae463
Merge utils/Types.h with common/Types.h ( #16445 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-05-02 22:21:51 +08:00
1a29337e6b
Print unflushed segmentIDs instead of whole SegmentInfo ( #16301 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-29 19:13:47 +08:00
f18b3c44ee
Print msgID instead of msg ( #16268 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-29 19:11:47 +08:00
6440c0dd98
Add log for getting IndexNode's mertrics ( #16747 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-04-29 19:07:47 +08:00
9e09434a4b
Remove not used QueryService in querynode ( #16735 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-29 18:15:48 +08:00
2fe8677cbf
Enable dimension check in Proxy when create index request received ( #16718 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 18:01:49 +08:00
bb9ccbb7e2
Use a singleton kafka producer ( #16739 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-29 17:29:47 +08:00
5b2b917987
Benchmark go/cgo parquet payload readers ( #16736 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-29 15:59:47 +08:00
7dfab0aa13
Add unit test for case of failed to sync segments to shard leader ( #16712 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-29 15:51:49 +08:00
97757405de
Fix bug: wrong metrics name and forget to register metrics ( #16714 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-29 15:35:47 +08:00
93416d5524
Fix LoadBalance loads the same segment multiple times ( #16734 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-29 15:19:47 +08:00
2f49db9d3c
Implements remaining rest APIs ( #16387 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-04-29 15:15:47 +08:00
415a0e5978
Fix miss segments/DmChannel when query by nodeID ( #16719 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-29 14:11:47 +08:00
a38c84e7e7
Lock meta-replica during search/query ( #16703 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-29 13:37:47 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
68b1b82faf
Remove DataKV interface ( #16692 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 21:03:47 +08:00
21b5973253
Refine list import task api ( #16690 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-28 17:21:47 +08:00
2205b878b1
Refine log in query_shard.go ( #16693 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 17:13:48 +08:00
206b554d49
Sync the distribution of segments to shard leaders ( #16685 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-28 16:01:48 +08:00
ed51ffd50f
Fix Compact Entity ttl param ( #16665 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-28 11:51:47 +08:00
c0a73dbe98
Using a temporary dir instead of a specified dir within embed etcd test ( #16694 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-04-28 11:17:47 +08:00
7017756601
Validate the number of replicas for load collection/partitions request ( #16697 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-28 10:54:00 +08:00
0a953948af
Simplify monitoring metrics ( #16687 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-27 23:03:47 +08:00
3a6db2faeb
Fix handling segment change logic ( #16695 )
...
Dispatch segmentChangeInfo to ShardCluster leader
Hold segment remove before search is done
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-27 22:23:46 +08:00
bb6cd4b484
Disable the unit test for the case of filter out unavailable leaders of GetShardLeaders ( #16688 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-27 22:15:47 +08:00
be5d1f1243
Fix Datacoord ut when etcd is slow ( #16689 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-27 21:49:47 +08:00
056fd9aa17
Correct wrong words ( #16490 )
...
Signed-off-by: chenlifeng <chenlifeng0995@wezhuiyi.com>
Co-authored-by: chenlifeng <chenlifeng0995@wezhuiyi.com>
2022-04-27 19:51:46 +08:00
2879796b42
Validate partitionIDs & segmentIDs in query request ( #16684 )
...
* check partitions if they are released or unloaded
* check segments if their collection/partition(s) are released or unloaded
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-27 19:23:48 +08:00
35b7267edb
Add ut for task_policies ( #16663 )
...
See also: #16652
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-27 16:55:46 +08:00
68ef720d6b
Filter out the unavailable shard leaders ( #16677 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-27 16:27:46 +08:00
0bf176a7e1
Add SyncReplicaSegments interface for QueryNode ( #16672 )
...
Add SyncReplicaSegments for LoadCollection/LoadPartition child task
This API notifies related ShardCluster to record segment Loaded
Preventing the Etcd events is delayed, which will cause some segment not search
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-27 10:41:46 +08:00
b657b10eb5
Remove useless log ( #16624 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-27 10:01:47 +08:00
60f7fef3b3
Remove offline nodes from replica info after LoadBalanceTask done ( #16653 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-26 18:23:46 +08:00
a3f7bb5f2d
Remove the collectionID label from metrics ( #16571 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-26 17:29:45 +08:00
4855ba6fa3
Optimize Mac M1 support some tweaks to embedded Milvus ( #16650 )
...
issue: #16500
issue: #16621
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-26 14:21:46 +08:00
f75dedb317
Validate partitionIDs & segmentIDs in search request ( #16639 )
...
* check partitions if they are released or unloaded
* check segments if their collection/partition(s) are released or unloaded
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-26 14:19:45 +08:00
80ae6de323
Add comments for concurrency package ( #16654 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-26 11:33:45 +08:00
c1ff0cec8f
Change QueryCoord Log level ( #16590 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-26 11:29:54 +08:00
fbc7fe1cdc
Fix proxy unable to update cache ( #16646 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-26 11:27:53 +08:00
bc5e9ec929
Add list import tasks api ( #16605 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-25 17:37:46 +08:00
bb69e09736
Use fmt.Errorf instead of string concat in querynode/segment.go ( #16422 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-25 16:11:48 +08:00
7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment ( #16514 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-25 15:57:47 +08:00
1682dc49fc
Fix unit test timeout ( #16537 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-04-25 14:49:46 +08:00
b99b65c26e
Add segment reference count and handles change info in ShardCluster ( #16620 )
...
Resolves #16619
Add reference count for each search/query request
For SegmentChangeInfo
- Wait all segments in OnlineList to be loaded
- Add handoff event into pending list
- Wait all segments in OfflineList is not used (reference count = 0)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-25 11:51:46 +08:00
4ef2df8cb9
Fix an issue where query coord calls MinioChunkManager during init ( #16629 )
...
It causes Milvus not runnable with local storage.
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:11:46 +08:00
e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated ( #16609 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-25 11:07:47 +08:00
65a9e01a8f
Use AssertInfo instead of assert to ensure compiling ( #16533 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-25 10:27:46 +08:00
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
5c66300408
Always add the shard leader into the nodes list of shard ( #16616 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-24 17:37:46 +08:00
04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering ( #16520 )
...
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
2022-04-24 16:43:45 +08:00
b82f6a1a0a
Fix GetReplicas() returns all nodes in ShardReplicas ( #16597 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-24 11:35:45 +08:00
242f08fb67
Refine import task related logic ( #16601 )
...
issue: #16600
/kind bug
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-24 11:29:45 +08:00
2022a361c0
Enable ZSTD compression for pulsar ( #16014 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-24 10:25:43 +08:00
0cec1b9448
Fix bulkload bug ( #16591 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-22 18:43:43 +08:00
3f5733b738
Return the local variable directly, make RVO enabled ( #16201 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-22 17:43:42 +08:00
804fbb6e27
Fix panic: Set Base of GetReplicas request ( #16578 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-22 14:47:42 +08:00
bd29518b6e
Remove debug log with no meaning ( #16583 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-22 13:11:42 +08:00
6de1d223a6
Remove debug log in querynode segment/node detector ( #16572 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-22 11:07:42 +08:00
a6a3b69d91
Report bulk load state ( #16555 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-04-21 21:37:42 +08:00
91e84ffedf
fix: not create user root if exists, optimize log & imports ( #16554 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-04-21 19:57:42 +08:00
6f75d02c65
Disable knowhere logging for embedded Milvus and some other tweaks ( #16496 )
...
/kind enhancement
issue: #15711
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 17:23:46 +08:00
93777e0c9d
Handle flowgraph releasing properly ( #16169 )
...
See also: #15846
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-20 16:45:41 +08:00
Jiquan Long
exia
congqixia
Xiaofan
Ten Thousand Leaves
yah01
Letian Jiang
Bingyi Sun
cai.zhang
xige-16
Enwei Jiao
bigsheeper
groot
shaoyue
jaime
Eng Zer Jun
Cai Yudong
zhenshan.cao
XuanYang-cn
Aivin V. Solatorio
codeman