milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
21,509 Commits
152 Branches
122 Tags
588 MiB
Commit Graph

1736 Commits (f441ccdbe9d8c0fb3f55e7030456371c405434d1)

Author SHA1 Message Date
congqixia 10acdbbe8e
enhance: free CString in InitTraceConfig (#30055) 
`C.CString` result needs to be freed after usage

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-17 15:15:03 +08:00
congqixia c0f0548702
fix: use SafeChan preventing close channel multiple times (#30022) 
See also #29935

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-16 17:34:54 +08:00
Bingyi Sun e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
wayblink 1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641) 
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-11 19:12:51 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
Cai Yudong cb9d9ec0f0
enhance: Correct sampleFraction's type to float (#29810) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2024-01-10 13:18:50 +08:00
yihao.dai 3d07b6682c
feat: Add import reader for numpy (#29253) 
This PR implements a new numpy reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-08 19:42:49 +08:00
yihao.dai 156a0dd450
feat: Add import reader for Parquet (#29618) 
This PR implements a Parquet reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-07 19:38:49 +08:00
yihao.dai 23183ffb0f
feat: Add import reader for json (#29252) 
This PR implements a new json reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-05 18:12:48 +08:00
smellthemoon 1c1f2a1371
enhance:change some logs (#29579) 
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-05 16:12:48 +08:00
yihao.dai 3561586edf
feat: Add import reader for binlog (#28910) 
This PR defines the new import reader interfaces and implement a binlog
reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-05 11:48:47 +08:00
Jiquan Long 3f46c6d459
feat: support inverted index (#28783) 
issue: https://github.com/milvus-io/milvus/issues/27704

Add inverted index for some data types in Milvus. This index type can
save a lot of memory compared to loading all data into RAM and speed up
the term query and range query.

Supported: `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOL`
and `VARCHAR`.

Not supported: `ARRAY` and `JSON`.

Note:
- The inverted index for `VARCHAR` is not designed to serve full-text
search now. We will treat every row as a whole keyword instead of
tokenizing it into multiple terms.
- The inverted index don't support retrieval well, so if you create
inverted index for field, those operations which depend on the raw data
will fallback to use chunk storage, which will bring some performance
loss. For example, comparisons between two columns and retrieval of
output fields.

The inverted index is very easy to be used.

Taking below collection as an example:

```python
fields = [
		FieldSchema(name="pk", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
		FieldSchema(name="int8", dtype=DataType.INT8),
		FieldSchema(name="int16", dtype=DataType.INT16),
		FieldSchema(name="int32", dtype=DataType.INT32),
		FieldSchema(name="int64", dtype=DataType.INT64),
		FieldSchema(name="float", dtype=DataType.FLOAT),
		FieldSchema(name="double", dtype=DataType.DOUBLE),
		FieldSchema(name="bool", dtype=DataType.BOOL),
		FieldSchema(name="varchar", dtype=DataType.VARCHAR, max_length=1000),
		FieldSchema(name="random", dtype=DataType.DOUBLE),
		FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
```

Then we can simply create inverted index for field via:

```python
index_type = "INVERTED"
collection.create_index("int8", {"index_type": index_type})
collection.create_index("int16", {"index_type": index_type})
collection.create_index("int32", {"index_type": index_type})
collection.create_index("int64", {"index_type": index_type})
collection.create_index("float", {"index_type": index_type})
collection.create_index("double", {"index_type": index_type})
collection.create_index("bool", {"index_type": index_type})
collection.create_index("varchar", {"index_type": index_type})
```

Then, term query and range query on the field can be speed up
automatically by the inverted index:

```python
result = collection.query(expr='int64 in [1, 2, 3]', output_fields=["pk"])
result = collection.query(expr='int64 < 5', output_fields=["pk"])
result = collection.query(expr='int64 > 2997', output_fields=["pk"])
result = collection.query(expr='1 < int64 < 5', output_fields=["pk"])
```

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-31 19:50:47 +08:00
cai.zhang c45f8a2946
fix: Import data from parquet file in streaming way (#29514) 
issue: #29292

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-27 15:30:46 +08:00
XuanYang-cn 7a6aa8552a
fix: add back existing datanode metrics (#29360) 
See also: #29204

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-22 14:20:43 +08:00
congqixia f699be79f7
fix: grpc client check session skipped due to role not match (#29356) 
Related to #28815

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-21 10:12:51 +08:00
wei liu e41fd6fbde
enhance: Move proxy client manager to util package (#28955) 
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-20 19:22:42 +08:00
wayblink 2274aa3b50
fix: bulkinsert binlog didn't consider ts order when processing delta data (#29163) 
#29162

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-14 14:36:40 +08:00
Bingyi Sun ad866d2889
feat: integrate storagev2 into index build process (#28995) 
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-13 17:24:38 +08:00
wei liu fe1eeae2aa
enhance: Use mockery to replace manual mock code (#29074) 
issue: #29043
This PR remove mannul mock code for proxy and data coord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-13 10:46:44 +08:00
cai.zhang 49b8657f95
enhance: Support implicit type conversion for parquet (#29046) 
issue: #29019

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-12 16:14:44 +08:00
congqixia 1fe5f12bd5
enhance: Add client connect wrapper to keep connection alive (#29058) 
See also #29057
Add wrapper to maintain client&connection
When reset operation is needed, `Close` method shall wait until all
on-going request return

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-11 17:20:38 +08:00
cai.zhang 2b05460ef9
enhance: Make import-related error message clearer (#28978) 
issue: #28976

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-08 10:12:38 +08:00
wayblink 6736f65345
feat: skip some empty ttMsg in Datanode flowgraph (#28756) 
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-07 01:00:37 +08:00
yihao.dai d26b563a8b
feat: Define import API and metadata (#28731) 
Define the new rpc and metadata for ImportV2.

see also: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-04 19:56:35 +08:00
Bingyi Sun 45e6801ce4
feat: Add checker activation service interfaces (#28850) 
issue: #28610

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-04 17:38:37 +08:00
cai.zhang f5f4f0872e
enhance: Support importing data with parquet file (#28608) 
issue: #28272

Numpy does not support array type import. 
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-29 20:52:27 +08:00
cai.zhang 1b7a503f89
enhance: Revert import support csv format (#28760) 
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-28 14:32:27 +08:00
cai.zhang c29b60e18e
enhance: Support Array DataType for bulk_insert (#28341) 
issue: #28272 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-27 13:50:27 +08:00
MrPresent-Han fc30d291be
fix createCollection failed occasionally (#28592) (#28712) 
fix: create collection seldom failure #28592

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-27 11:10:25 +08:00
wayblink da339535d5
enhance: Merge flowgraph goroutines into 1 (#28654) 
/kind enhancement
#24826

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-23 19:52:25 +08:00
smellthemoon 73f2bab454
enhance:add some log when create client and get component states (#28160) 
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-22 09:12:22 +08:00
PowderLi c238bff9fb
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28580) 
/kind bug
to #28579 #28504

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-21 10:20:22 +08:00
Bingyi Sun d7145e2c06
enhance: Update golangci_lint version (#28535) 
Update golangci lint and fix some warnings

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-21 10:04:21 +08:00
PowderLi a1c505dbd5
add internal storage metrics (#28278) 
/kind improvement
issue: #28277

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:22:25 +08:00
XuanYang-cn 40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
smellthemoon 0aa90de141
Reduce the goroutine in flowgraph to 2 (#28233) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-11-13 10:50:17 +08:00
wei liu bce1054f92
Fix retry when proxy stopped (#28264) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 18:58:21 +08:00
groot 3f6b203018
Fix bulkinsert bug that segments are compacted after import (#28192) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-07 15:14:26 +08:00
wei liu 5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
wei liu da41a5b51e
fix check grpc error logic (#28182) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 11:54:18 +08:00
Xiaofan da19e49daf
Support purge old session for standalone (#28184) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-11-06 21:21:42 +08:00
wei liu 68a86471ba
fix grpc client retry on node server not match error (#28169) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 23:42:16 +08:00
wayblink 00ae019ff0
Use go 1.20 csv_reader to keep milvus go=1.18 limitation (#28080) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-11-03 10:40:16 +08:00
wei liu ecec5dfcfd
fix retry on offline node (#28079) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 10:14:16 +08:00
groot abd5b199cc
Bulkinsert support pure list json (#27990) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-11-01 19:02:13 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
KumaJie e88212ba4b
Add CSV file import function (#27149) 
Signed-off-by: kuma <675613722@qq.com>
Co-authored-by: kuma <675613722@qq.com>
 2023-10-31 22:47:23 +08:00
yah01 9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
Enwei Jiao 4a33391b8f
rename createindex (#27903) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-27 10:12:14 +08:00
Filip Haltmayer 6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
zhagnlu 6060dd7ea8
Add chunk manager request timeout (#27692) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-23 20:08:08 +08:00
SimFG 9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
congqixia bcbe98aba1
Add querynode client wrapper and avoid grpc in standalone mode (#27781) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-19 11:10:07 +08:00
jaime ac2d1bb5c2
Support receive signals from parent process (#27756) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-18 20:20:11 +08:00
yihao.dai 49b3a12804
Return newly defined merr instead of grpc unimplemented err (#27751) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-18 15:32:11 +08:00
congqixia 2f201c25e2
Remove deprecated io/ioutil usage (#27747) 
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
 2023-10-17 20:32:09 +08:00
wei liu 7aa862c0ea
fix retry on unimplemented rpc error (#27639) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-16 10:50:09 +08:00
jaime ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
yah01 be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
Jiquan Long e4f73cc805
Add host & enable_disk to session (#27507) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-08 20:05:31 +08:00
yah01 8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
Jiquan Long 5c1abfa2cc
Print the server id when active-standby switch (#27119) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-07 10:01:31 +08:00
Jiquan Long 0f14d18201
Optimize the codec code of session (#27360) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-01 10:33:30 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
SimFG c9653b1683
Add some log and improve TestSessionProcessActiveStandBy test case (#27403) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-28 09:35:27 +08:00
XuanYang-cn 5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
yah01 6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
wayblink 7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
jaime 7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
XuanYang-cn 676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
wei liu 9433a24f5d
fix component not exit when liveness check failed (#27236) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-22 19:13:25 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
SimFG 26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia 16b35e07b3
Fix `TestSessionSuite/TestKeepAliveRetryActiveCancel` unit test logic (#27231) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-20 18:59:23 +08:00
yah01 9baff1b81c
Rename error to match the nameing rules (#27214) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:59:24 +08:00
yah01 b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
congqixia cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai 4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
Enwei Jiao 8e17bc3013
Use merr to handle segcore errors (#27137) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-18 14:07:22 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
yah01 168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yah01 7bb20fa394
Fix the double nil return values from RPC call (#27101) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:01:19 +08:00
congqixia c32a9a1d4a
Fix grpcclient use needReset as needRetry for cancel&timeout code (#27091) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-14 17:47:26 +08:00
congqixia f0d0651989
Do not reset connection immediately if grpc code is `Canceled` or `DeadlineExceeded` (#27014) 
We found lots of connection reset & canceled due to recent retry change
Current implementation resets connection no matter what the error code is
To sync behavior to previous retry, skip reset connection only if cancel error happens too much.

Also adds a config item for minResetInterval for grpc reset connection

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-13 15:01:18 +08:00
yihao.dai dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
yah01 00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
Jiquan Long 802a6a86c0
Fix printing too many warnings (#27015) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-12 13:39:17 +08:00
aoiasd e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
congqixia adfb5298c6
Refine `TestSessionProcessActiveStandBy` unit test logic (#26980) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-11 18:13:17 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn 7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
SimFG 0901b76732
Avoid the panic when the status of rpc response is nil (#26910) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-07 19:23:15 +08:00
yiwangdr 337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
SimFG 28681276e2
Improve the retry of the rpc client (#26795) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-06 17:43:14 +08:00
wei liu 0e2085b77f
fix dc standby to active (#26810) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-06 10:41:49 +08:00
Enwei Jiao fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
congqixia 1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
chyezh 0530fd80c9
[Fixup] remove nats from default (#26791) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-04 10:01:04 +08:00
yah01 3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
groot 1b1bafaff1
Bulkinsert read varchar by batch (#26199) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-08-30 14:23:06 +08:00
yah01 dd4bc5b6a0
Fix data race in gRPC client (#26574) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-28 18:26:28 +08:00
congqixia 145387fdcb
Bump proto go-api to v2.3.0 (#26561) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-23 20:18:23 +08:00
jaime b82fc8d6e7
[skip e2e] Add connection log for etcd and grpc (#26536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-22 21:20:24 +08:00
yah01 bed034a44b
Protect DataCoord from calculating segment lines by stale log entries num (#26523) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-22 14:16:22 +08:00
chyezh 85886f039e
[Fixup] msg context access data race when msgdispatcher (#26463) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-08-21 19:16:22 +08:00
XuanYang-cn 08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
wayblink 71133e5eaa
Check and reset if grpc client serverID mismatch with session (#26448) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-18 22:50:20 +08:00
yihao.dai 63b86b32a6
Add server id validation interceptor (#26395) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-17 20:20:20 +08:00
wei liu 74133a3996
refine retry on grpc (#26360) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-16 19:36:18 +08:00
SimFG 8b54914287
Add go.mod file for the milvus/pkg package (#26359) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-16 14:49:32 +08:00
Enwei Jiao 78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
Gao b6fcbb0998
Support ScaNN index (#26099) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-11 14:21:29 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
congqixia 2b367b6bb0
Fix sessionutil Liveness check blcok in watch forever (#26248) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-10 14:07:16 +08:00
congqixia e13b6b88eb
Replace deprecated grpc WithInsecure option (#26226) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 13:37:16 +08:00
congqixia 767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 10:05:14 +08:00
wei liu 0e103efc5d
disable retry in grpc connect and grpc sendMsg (#26095) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-03 15:23:07 +08:00
congqixia 7dfc8fbf0a
Fix data race on keepAliveCancel (#26087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 18:55:07 +08:00
congqixia 8b11636e72
Cancel previous ctx for session retry keepalive (#26050) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-02 12:09:05 +08:00
groot c2693ea284
Fix a crash bug of bulkinsert for dynamic field (#25982) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-28 15:25:03 +08:00
XuanYang-cn 84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
zhagnlu 833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-27 19:49:02 +08:00
groot a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
SimFG 69d274d233
Improve the operation log (#25589) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-14 16:08:31 +08:00
Enwei Jiao 66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
wayblink 587237a3c9
Fix dead loop in session (#25451) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 18:02:29 +08:00
groot 96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
yah01 cd29b863d0
Fix data race in session (#25354) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 14:52:25 +08:00
wayblink b7ecb7f56b
Disable retryKeepAlive when LivenessCheck's Context close (#25161) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-27 17:08:45 +08:00
xige-16 33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
wayblink b752a29995
Add timeout for keepalive in session (#25077) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-26 12:30:44 +08:00
jaime 18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
yihao.dai 9f5756f16e
Add cluster validation interceptor to resolve the ip-reuse issue (#25001) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-21 11:36:46 +08:00
SimFG 0c3f92d7d7
Improve the panic code about the rootcoord/session/rocksmq (#24859) (#25024) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-06-21 11:24:42 +08:00
wayblink bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
yiwangdr 4387f36897
make etcdKV private (#24778) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-13 10:52:38 +08:00
congqixia 41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01 ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
Enwei Jiao d3af451d92
Upgrade golangci-lint (#24707) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-06-07 19:34:36 +08:00
congqixia d0c2fa5d19
Fix retryKeepAlive assertion panic (#24667) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-07 10:08:36 +08:00
chyezh f97127ae55
add nats mq wrappers (#24445) 
bug fixup, configurable natsmq, add unittest, pass e2e.



move natsmq to pkg project

Signed-off-by: chyezh <ye.zhen@zilliz.com>
Co-authored-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-07 10:00:37 +08:00
xige-16 732fe54775
Support partition Key (#24047) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-06 10:24:34 +08:00
wayblink 5fb5b072ae
Retry keepalive when keepalive channel close (#24581) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-01 16:14:35 +08:00
groot 3022e37298
Fix bug of bulkinsert for dynamic field (#24569) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-01 14:14:31 +08:00
groot 5358da2a10
Fix a crash bug for backup/restore (#24529) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:45:30 +08:00
groot 0d0e97ab7f
Fix crash issue of bulkinsert by invalid numpy array file (#24480) (#24528) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:43:39 +08:00
groot 3cb75997a3
Fix a bug of bulkinsert dynamic schema (#24405) (#24527) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-31 14:41:41 +08:00
xige-16 fe18109aab
Clean tmp disk data when start milvus (#24404) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-05-26 15:07:26 +08:00
wei liu ce03248b1a
simplify logic circle in search (#24110) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-23 16:01:26 +08:00
groot 97cd0409e4
Support dynamic field for bulkinsert (#24265) (#24304) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-23 14:17:25 +08:00
Jiquan Long bd343550a5
Support to manage connections (#24224) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-05-19 12:51:23 +08:00
MrPresent-Han 7744573d3d
support parms for import maxfilesize(#24191) (#24192) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-05-18 18:59:27 +08:00
groot b89f06dd19
Support JSON for bulkinsert (#24130) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-05-18 16:33:23 +08:00
congqixia 73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia 74bba2320a
Fix session stop/goingStop stuck after connection lost (#24131) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 14:51:22 +08:00
Enwei Jiao a933b595c4
Add trace log for grpc client (#23951) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-05-09 10:10:41 +08:00
wayblink 899702f13c
Implement GetIndexStatistics interface (#23603) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-06 10:34:39 +08:00
Enwei Jiao 967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
wei liu cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
cai.zhang 43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
jaime c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
congqixia 424570b2b8
Remove annoy related unit tests (#23229) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-05 08:14:28 +08:00
Cai Yudong ef63e64ded
Remove ANNOY index type (#23189) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-04 16:30:27 +08:00
Cai Yudong 9ad6d9f1a0
Rename RAFT_IVF_FLAT/RAFT_IVF_PQ to GPU_IVF_FLAT/GPU_IVF_PQ (#23194) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-04 11:44:26 +08:00
yah01 7da870f512
Remove useCustomConfig and simpilify the session type (#23166) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-03 20:10:24 +08:00
smellthemoon ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
MrPresent-Han afd874b736
enhance segment balance by considering global rowCount(##22914) (#23056) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-04-03 14:16:25 +08:00
Cai Yudong 7612c75c47
Let RAFT_IVF_PQ param accept m=0 (#23134) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-31 11:22:22 +08:00
yah01 75737c65ac
Refine error handle of QueryCoord (#23068) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-31 10:54:29 +08:00
yihao.dai a182594bf5
Add FlushAll support (#22942) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-30 11:28:25 +08:00
Xiaofan 653d9ecf91
Change RocksMQ page size to 64M (#23078) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-29 17:08:04 +08:00
Cai Yudong 3febb5e45a
Remove gpu mode param check (#23074) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-29 14:24:01 +08:00
smellthemoon 846de72fb3
fix to reduce memory (#23023) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-28 21:36:06 +08:00
Xiaofan d0510210df
Add config for disable BF load (#22996) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-28 20:06:10 +08:00
aoiasd 5d172d0f4f
Refine errors of datanode (#22852) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-28 19:04:00 +08:00
zhenshan.cao d55c860383
Align the maximum dim of the diskann index and collection (#23039) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-03-28 00:06:00 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan 5a914aedf3
Make Datacoord TT to be configurable (#22956) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-24 22:24:04 +08:00
Cai Yudong 8aebc6f3b7
Remove faiss GPU index support (#22966) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-24 17:53:58 +08:00
Jiquan Long 0be96ab610
Check if all columns aligned with same num_rows (#22968) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-24 17:09:58 +08:00
smellthemoon 3dae84f065
Use merr in proxy (#22904) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-24 15:27:58 +08:00
Gao 8b3e5189e1
Add querynode plugin (#22379) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-03-24 15:21:59 +08:00
congqixia 1b4e29e1ee
Fix copylocks for EnvSource (#22948) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-23 19:26:02 +08:00
SimFG e8f8c1b445
Check the collection num when creating the collection (#22946) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-23 16:47:57 +08:00
congqixia 31c22e5d24
Fix NodeID rw race (#22945) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-23 11:45:57 +08:00
yihao.dai f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
Enwei Jiao fd974df09f
OTEL integration in cpp (#22175) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-21 16:31:57 +08:00
yah01 532d12c4cf
Fix the code lost with multi errors (#22877) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-21 14:07:57 +08:00
smellthemoon 4c603cd02c
Fix upsert msgid (#22839) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-21 14:06:01 +08:00
congqixia 0317e0458b
Use Set[string] instead of map[string]interface{} ServerType (#22853) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-20 16:21:57 +08:00
yihao.dai 1f718118e9
Dynamic load/release partitions (#22655) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-20 14:55:57 +08:00
MrPresent-Han 77c9e33e70
support dml channel balancer on datacoord (#22324) (#22377) (#22692) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-03-20 10:01:56 +08:00
Jiquan Long 8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
yah01 3d8f0156c7
Refine scheduler & executor of QueryCoord (#22761) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-16 17:43:55 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-14 23:21:56 +08:00
congqixia 732986aa04
Remove fmt.Print from internal package (#22722) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-14 17:36:05 +08:00
groot 6f6bd98c27
Verify vector float data for bulkinsert and insert (#22728) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-03-14 14:03:58 +08:00
yah01 e1cb307690
Refine RootCoord services error handle (#22689) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-13 14:29:53 +08:00
congqixia af6f0d5aa5
Make merr.Error(status) read old error code (#22698) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-10 22:37:53 +08:00
yah01 e4120d1671
Upgrade proto to add error code field of Status (#22656) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-09 18:49:51 +08:00
shaoyue 32581e6452
Support aliyun oss as object storage with ak or IAM (#22376) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-03-09 16:33:52 +08:00
yah01 90a5aa6265
Refine errors, re-define error codes (#22501) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-09 15:47:52 +08:00
wei liu 1a1a5caafa
update shard leader cache periodically (#22595) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-03-08 15:13:51 +08:00
Enwei Jiao f1a60b295c
Support multi yaml files (#22583) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-07 18:23:51 +08:00
jaime d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
Enwei Jiao b21395f487
Forbid dmlChannelNum auto refresh (#22550) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-03 15:21:48 +08:00
congqixia ea4ab9e4a4
Replace ErrorList with multiErrors (#22532) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-03 14:59:49 +08:00
Gao aa19243c22
Fix diskann param parse bug (#22498) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-03-02 19:59:50 +08:00
congqixia 51cb8590a1
Use errors.Wrap and Combine so retry error could check `errors.Is` (#22520) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-02 17:25:48 +08:00
Ten Thousand Leaves 74d158e149
Fix enableAutoCompaction default value (#22254) 
/kind bug

issue: #22253

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2023-03-02 07:21:48 +08:00
yah01 7478e44911
Support using mmap to load data (#22052) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-01 18:07:49 +08:00
congqixia fa5c33975e
Fix reset connection data race (#22483) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-01 11:21:47 +08:00
yah01 21eea923cc
Add util to combine multiple errors so everyone checkable (#22473) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-28 19:35:47 +08:00
yah01 60cd548bf5
Rename concurrency package to conc (#22453) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-28 14:19:47 +08:00
congqixia 608615e5bd
Add background health check if ctx err returned (#22439) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-28 14:11:47 +08:00
Xiaofan 256aaa1944
Change Etcd session timeout to 20s (#22401) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-28 11:13:47 +08:00
Xiaofan fa101e9751
Change RockMQ page size to 256MB (#22432) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-28 10:45:47 +08:00
yah01 167581c08a
Refine Pool to improve the ease of use (#22412) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 18:07:48 +08:00
wayblink 416866e42f
Add memory usage too large sync policy (#22244) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-27 17:47:51 +08:00
yah01 b758c305a7
Make Future generic (#22411) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-27 12:07:47 +08:00
Enwei Jiao 697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
zhenshan.cao e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
SimFG bddda5bdb2
Only print the file content when log the hook.yaml (#22361) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-02-23 15:33:45 +08:00
Enwei Jiao 0851e05014
Gernate milvus.yaml by code (#22003) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-23 11:37:46 +08:00
SimFG 7e19a924b7
Add the rbac for the GetLoadingProgress and GetLoadState api (#22320) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-02-22 20:37:45 +08:00
Cai Yudong d4e0b6e91b
Remove unused index param check (#22271) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-02-21 17:36:26 +08:00
wayblink b7c0d12d82
Skip file with prefix '.' may exist in filesystem (#22138) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-21 10:54:29 +08:00
bigsheeper f4e7b246f2
Fix msgdispatcher concurrency issue and add prometheus metrics (#22174) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-02-16 14:58:34 +08:00
congqixia f2575e5fa8
Add unconvert & durationcheck linters and fix issues (#22161) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-15 17:22:34 +08:00
aoiasd 148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
wayblink 77a27c6dfc
Support print log to file and console at the same time (#21946) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-14 16:56:34 +08:00
SimFG c2a49d5a0b
Add error log and metrics for the hook and privilege interceptor (#22111) (#22137) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-02-13 16:50:33 +08:00
bigsheeper d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel (#21917) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-02-13 16:38:33 +08:00
jaime a2435cfc4f
[Cherry-Pick]Add a segment metrics for count number of binlog files (#22103) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-13 10:24:33 +08:00
wei liu d078441522
support rg api rbac (#22097) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-10 10:54:33 +08:00
Xiaofan d3f8948c17
Fix embedded milvus (#22077) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-09 22:06:32 +08:00
wei liu d085abbd56
fix load collection with rg (#22083) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-02-09 16:24:31 +08:00
yah01 b1f31da77a
Fix activate standby server ignores all errors (#22073) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-09 15:24:31 +08:00
Enwei Jiao 44154a828c
Forbid updating some configs (#22047) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-09 14:28:31 +08:00
groot c21e11eb8a
Fix datanode crash bug for bulkinsert (#22041) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-02-08 14:06:31 +08:00
Jiquan Long a548ce1cf1
Change default max shards num to 64 (#22016) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-02-07 14:43:57 +08:00
jaime baa035885d
Modify default value of some configurations (#22013) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-07 10:05:56 +08:00
congqixia 73b6c13245
Expand singleflight to Get and make writeEvent sync (#21993) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-06 15:27:54 +08:00
cai.zhang ae305a54bd
Set scheduler duration of index task can be configurable (#21944) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-05 17:21:53 +08:00
jaime 748a935a11
Modify segment seal policy with number of statslog files (#21957) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-03 17:07:52 +08:00
wayblink d41cc0b21b
Revoke session to only delete session key created by this node (#21935) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-02 16:37:52 +08:00