8438386395
fix pq distances cal bug ( #3876 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
c171daa184
let bitset nullptr it no deletion and no filter ( #3872 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
273d12f2da
Fix SearchTask::nq() ( #3559 )
...
* Fix field_name duplicate problem
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix SearchTask::nq
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix SearchTask
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix must_not query
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
6f45511023
validate binary index type ( #3869 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
b6134311e0
fix gpu check bug ( #3866 )
...
* fix gpu check bug
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* format code
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
1784233528
(db/snapshot): fix bug in metadata snapshot that handling CollectionCommit and PartitionCommit mappings ( #3864 )
...
Signed-off-by: peng.xu <peng.xu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
6939c60bef
reduce uid copy during search ( #3867 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
7bfcec642f
Set the bitset of searching ( #3858 )
...
Signed-off-by: cqy <yaya645@126.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
7785f44ef4
reduce insert memory copy ( #3842 )
...
* avoid memory copy for insert request
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
* fix unittest failure
Signed-off-by: groot <yihua.mo@zilliz.com>
* add more log
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
* add log
Signed-off-by: groot <yihua.mo@zilliz.com>
* add log
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
* refine code
Signed-off-by: groot <yihua.mo@zilliz.com>
* format code
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
cd70c122f1
catch-load-config-exception ( #3857 )
...
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
remove unused code
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
f168f45183
add more wal log ( #3850 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
ece04f3ccb
validate structured index type ( #3848 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-29 14:03:26 +08:00
8cd839ea3b
Add S3 storage support for Milvus ( #3874 )
...
* Modify S3 related code (unfinished)
* Delete unnecessary modification about AWS
* Modify config so as to complete unit test of S3
* Modify config so as to complete unit test of S3
* Complete unit test of S3
* Implement unit test of writing and reading DeletedDocs with segment writer / reader(unfinished)
* Temporarily stop using unittest/segment to test segment writer and reader
* Delete the directory unittest/segment and all files in the directory
* Refine code
* Complete unit test of writing and reading deleted docs file with SegmentWriter and SegmentReader
* Update DeletedDocsFormat.cpp with upstream code
* Refine code
* Submit unit test code to read and write vector index with segment writer and reader
* Hide the code of filesystem in order to support S3 and other storage strategies
* Delete code of index r/w unit test;
Complete unit test of writing and reading bloom filter with segment writer / reader
* Refine code
* Modify the code style to pass Cpplint and clang-format
* Delete the code to disable S3
* Modify the code to disable S3
2020-09-26 17:21:41 +08:00
0c06e2a155
Use bash by default in the build environment Docker container ( #3879 )
...
* Add entrypoint.sh
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
* Add entrypoint.sh
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
* Update docker-compose.yaml
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
* Update milvus build environment dockerfile
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
* Update milvus build environment dockerfile
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
* Update milvus build environment docker entrypoint file
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2020-09-26 10:48:20 +08:00
b03362bbbd
[skip ci] update install from code ( #3855 )
...
* [skip ci] update install from code
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
* [skip ci] update install from code
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-24 09:18:50 +00:00
a4cde384d2
Fix the multi-gpus building index enable problem ( #3845 )
...
Signed-off-by: cqy <yaya645@126.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
97265aa68b
avoid insert hangs ( #3841 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
0e7bd80443
remove the flush step in drop collection request processing ( #3840 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
a9242b72fe
Fix the error message should be updated if search on index IVF_SQ8_HYBRID with CPU ( #3820 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
40c37cb94d
Limit the number of insert requests processing concurrently ( #3830 )
...
* Limit the number of insert requests processing concurrently
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* code format
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* change the variable name
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* change comment
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
ca60fa2cc8
update oatpp md5 ( #3832 )
...
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
9baf7bac14
fix wal recovery issue ( #3818 )
...
* fix wal recovery issue
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
e5f763f30c
add hidden config engine.max_partition_num ( #3817 )
...
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
962d0828b7
use multi-segments operation to flush ( #3804 )
...
* use multi-segments operation to flush
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
0571fe18b9
Fix web start fail for IP family ( #3778 )
...
* Fix web start fail for ip family
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* update oatpp md5
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* Add more detail case
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-23 13:08:11 +08:00
bbc46b7d55
when the topk of gpu search >2048 ,turning into the cpu search ( #3640 )
...
* search task
Signed-off-by: cqy <yaya645@126.com>
code refactoring
Signed-off-by: cqy <yaya645@126.com>
* fix sq8h pass
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
* modify the range of param 'ef' in HNSW
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-19 16:42:04 +08:00
836a247b6a
Fix the crash caused by empty index param ( #3803 )
...
* fix crash caused by empty index param
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
* fix C++ SDK examples
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
d1ab5bb2c7
fix the limit of nprobe and nlist ( #3797 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
2f4207acb2
Improve MergeTopkToResultSet performance ( #3788 )
...
* Fix WebServer README.md
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Add changelog
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix cpplint
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix Readme.md
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix MergeTopkToResultSet performance problem
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
ee4789cc5f
avoid snapshot warning during deleting entity ( #3798 )
...
* avoid snapshot warning
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
b383748870
Fix the bug of file not close when check sum failed. ( #3783 )
...
* Fix the bug of file not close when check sum failed.
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* change the log level to error
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* fix cpp lint problem
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
b29954cd9e
avoid dead circle when merge failed ( #3787 )
...
* avoid dead circle when merge failed
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
35a5631698
add adaptive merge strategy ( #3780 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
6c314348e9
Fix webserver readme for 0.11.0 ( #3765 )
...
* Fix WebServer README.md
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Add changelog
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
* Fix cpplint
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-18 18:59:24 +08:00
caa251f30f
hold FaissIVFQuantizerPtr in IVFSQHybrid ( #3770 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
7a4e87fcce
modify merge strategy ( #3757 )
...
* modify merge strategy
Signed-off-by: groot <yihua.mo@zilliz.com>
* clean code
Signed-off-by: groot <yihua.mo@zilliz.com>
* typo
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
1f5850a4a7
close lockfile when server stop ( #3737 )
...
* close lockfile when server stop
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
* fix something error
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
3b8a69a3e2
#3756 ignore deleted docs file if it doesnt exist ( #3766 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
b5ffb46d21
fix trace level log files missing ( #3769 )
...
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
2a0bf43445
New operation named MultiSegmentsOperation support multi-segments during flush ( #3733 )
...
* Add multiSegmentsOpera
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* rish test
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* Add new operation MultiSegmentsOperation
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* Recoveray ut itls
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
* fix judeg bug
Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
ade8bb3f23
fix config usage ( #3747 )
...
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
41dbcf7986
(db/snapshot): fix GC PartitionCommit ( #3750 )
...
Signed-off-by: peng.xu <peng.xu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
b62a05cb9b
Improve code of dabloom ( #3694 )
...
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
52b0181e89
Apply more stale check rules for Operations ( #3721 )
...
* (db/snapshot): Update Operations to handle stale errors
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): add some print in ut
Signed-off-by: peng.xu <peng.xu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
82711d6065
rename segment_row_count to segment_row_limit ( #3724 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
dd31d87c4c
GET_PAGE_ENTITY sometimes failed in test_web ( #3735 )
...
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
5f9a7405f6
[skip ci] disable two compact cases ( #3731 )
...
Signed-off-by: zw <zw@milvus.io>
Co-authored-by: zw <zw@milvus.io>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
8a4d8c4d51
add some check to avoid server crash ( #3715 )
...
* add some check to avoid server crash
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
* rm useless code
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
3b29ae776f
fix delete entity performance issue ( #3667 )
...
* fix delete entity performance issue
Signed-off-by: groot <yihua.mo@zilliz.com>
* force flush for delete
Signed-off-by: groot <yihua.mo@zilliz.com>
* improve delete performance
Signed-off-by: groot <yihua.mo@zilliz.com>
* fix delete performce issue
Signed-off-by: groot <yihua.mo@zilliz.com>
* fix hang
Signed-off-by: groot <yihua.mo@zilliz.com>
* segment row count min limit
Signed-off-by: groot <yihua.mo@zilliz.com>
* merge 0.11.0
Signed-off-by: groot <yihua.mo@zilliz.com>
* fix python test
Signed-off-by: groot <yihua.mo@zilliz.com>
* fix python test failed
Signed-off-by: groot <yihua.mo@zilliz.com>
* add log
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
be67ef32f9
(db/snapshot): Implement FlushableMappingField for better metadata snapshot ( #3655 )
...
* Update metadata for mappings
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 2
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 3
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 4
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 5
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 6
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* Update metadata for mappings 7
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): optimize PartitionCommit metadata implementation
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): optimize PartitionCommit metadata implementation
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): update mock logic
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): small change
Signed-off-by: peng.xu <peng.xu@zilliz.com>
* (db/snapshot): small change
Signed-off-by: peng.xu <peng.xu@zilliz.com>
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
2020-09-17 14:15:56 +08:00
XuanYang-cn
shengjun.li
yukun
groot
chen qingxiang
XuPeng-SH
cqy123456
Wang Xiangyu
WeiFeng Lai
quicksilver
BossZou
Jin Hai
del-zhenwu