milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
20,055 Commits
141 Branches
117 Tags
549 MiB
Commit Graph

20055 Commits (f993b2913bf7f6d367600c7a28824818ff8e44b9)

Author SHA1 Message Date
congqixia f993b2913b
enhance: Reserve space of payload writer when serialize data (#33817) 
See also #33561 #33562

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-17 12:06:04 +08:00
wei liu 188ee433b9
enhance: Refine name rule check error msg (#33815) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:37:58 +08:00
cai.zhang 95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
wei liu b72026c859
enhance: Add rbac support on describe/alter database api (#33803) 
issue: #32707

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:07:57 +08:00
yihao.dai 1a9ab52f66
enhance: Ensure the idempotency of compaction task (#33872) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:09:57 +08:00
yihao.dai 8537f3daeb
enhance: Rename Compaction to CompactionV2 (#33858) 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
zhagnlu d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
zhagnlu e422168f09
fix: readd timestamp index because segment timestamp not ordered (#33856) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:48:01 +08:00
XuanYang-cn f67b6dc2b0
fix: DeleteData merge wrong data casuing data loss (#33820) 
See also: #33819

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 17:57:56 +08:00
nico b748d8af5d
test: add different datatype for some functions (#33869) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-06-14 17:51:57 +08:00
zhuwenxing a963afa0d3
test: refine load state check in restful test (#33861) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-14 16:22:10 +08:00
chyezh 9b69601dfb
fix: load operation when segment is on releasing (#31340) 
issue: #30857

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 15:35:56 +08:00
yihao.dai 86a36b105a
enhance: Tidy compaction executor (#33778) 
Move compaction executor to compaction pacakge.

issue: https://github.com/milvus-io/milvus/issues/32451

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-14 14:34:01 +08:00
shaoting-huang 0ecd694305
enhance: legacy code clean up (#33838) 
issue: #33839

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-14 14:25:56 +08:00
wei liu 4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
cai.zhang b69e9093c8
fix: Fallback field type when it isn't in request (#33832) 
issue: #33432

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-14 09:55:56 +08:00
sre-ci-robot f3e4cb0de9 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-06-14 00:01:27 +00:00
congqixia c689ef4822
fix: Remove eviction operations out of lock (#33834) 
See also #33823

`EvictCacheValueByFormat` may be block by on going `CASCacheValue` and
cause possible deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 21:15:55 +08:00
wayblink 08fcf3f62b
fix: Fix meta prefix overlap bug (#33830) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-13 19:27:57 +08:00
wei liu ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
ArenaSu 2dfa752527
doc: [skip-e2e] add comments for event manager (#33444) 
Add comments for event manager(internal/datanode/event_manager.go).

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:56:06 +08:00
ArenaSu db59249776
doc: [skip-e2e] add comments for meta table (#33403) 
Add commnets for meta table in internal/rootcoord/meta_table.go.

Signed-off-by: ArenaSu <704427617@qq.com>
 2024-06-13 17:54:09 +08:00
zhuwenxing ca1f7ab019
test: update import test case to support different dim (#33709) 
add test case for https://github.com/milvus-io/milvus/issues/33681

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-13 17:11:55 +08:00
congqixia 144ee269f2
fix: [skip e2e] Skip unstable integration test for master (#33824) 
See also #33716 #33823

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 16:53:55 +08:00
Yinzuo Jiang 7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 (#33141) 
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.

issue: #33142

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
 2024-06-13 15:27:58 +08:00
ThreadDao 9e99f960ab
test: add cases for go milvusclient delete (#33777) 
issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2024-06-13 14:43:57 +08:00
congqixia 512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs (#33562) 
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 11:15:56 +08:00
congqixia 9ab3058da2
fix: Prevent restart timetick sender creating ut datanode (#33790) 
See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 22:29:58 +08:00
congqixia b39dfc25dc
enhance: Use fastjson lib for unmarshal delete log (#33787) 
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 20:41:57 +08:00
chyezh b3ee1a4bd5
fix: lost DroppedAt field when DropCollection (#33766) 
issue: #33097

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:21:56 +08:00
cqy123456 be3559e022
enhance: proxy check hnsw with sparse is legal (#33697) 
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33653

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-12 20:13:55 +08:00
yihao.dai 9a3e4080f1
enhance: Add comment for channel cp updater (#33759) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-12 20:01:55 +08:00
congqixia b90999b741
fix: Compare config value then swap when caching param value (#33785) 
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 18:51:56 +08:00
Cai Yudong 9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-12 17:17:55 +08:00
Buqian Zheng 47b04ea167
enhance: support sparse cardinal hnsw index (#33656) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 16:57:55 +08:00
zhuwenxing 1697706ac0
test: [skip e2e] optimize get_image_tag_by_short_name function (#33780) 
Refactor the function to improve performance and readability. Instead of
making API requests to Docker Hub, the function now retrieves tags from
the Harbor registry. It also filters the tags based on the provided
architecture and selects the latest tag that matches the prefix. This
change enhances the efficiency of retrieving image tags by short name.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-12 14:27:56 +08:00
Buqian Zheng 8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector (#33209) 
issue: #29419

* sparse float vector to support raw data mmap

For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 10:09:55 +08:00
congqixia 323f8982ec
fix: [GoSDK] Use varchar when row field type is string (#33749) 
See also #33457

Also add `max_length` tag for specify varchar field max length attribute
from tagging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-11 23:39:55 +08:00
sre-ci-robot 64f673cb4e Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-06-11 12:01:03 +00:00
edward.zeng 17120afec6
test: [skip e2e] fix image tag (#33764) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2024-06-11 19:45:55 +08:00
edward.zeng 9a141b52f9
test: [skip e2e] fix the groovy for building docker images (#33763) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2024-06-11 19:15:54 +08:00
edward.zeng 4ed3b5e5c1
test: [skip e2e] fix the push order for milvus image (#33762) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2024-06-11 18:55:54 +08:00
coldWater 6b9901c59f
enhance: add a semaphore for CompactionExecutor (#33558) 
#33182

---------

Signed-off-by: coldWater <254244460@qq.com>
 2024-06-11 17:25:55 +08:00
Chun Han f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
Jiquan Long ecf2bcee42
enhance: speed up array-equal operator via inverted index (#33633) 
fix: #33632

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-11 14:13:54 +08:00
zhuwenxing fa26953168
test: fix unstable import case (#33726) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-11 14:03:54 +08:00
chyezh 2b7ee1968f
enhance: new messsage interface for log service (#33286) 
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-11 10:38:01 +08:00
chyezh 8ca5ced821
fix: async warmup will be blocked by state lock (#33686) 
issue: #33685

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-10 21:59:53 +08:00
yihao.dai b1d46eb34b
fix: Fix multiple vector fields import (#33723) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:57:54 +08:00
yihao.dai eb5d4de390
fix: Check if the import job exists (#33672) 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00