Commit Graph

159 Commits (038eebba4db629edd2498abf1b51009a4b91bf22)

Author SHA1 Message Date
aoiasd 4d8d34e370
Refine Rocksdb option (#24394) (#24503)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-06-15 15:12:39 +08:00
sre-ci-robot 7f7fe85592
Update knowhere commit (#24908)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-15 14:04:39 +08:00
sre-ci-robot ee2d458da4
Update knowhere commit (#24883)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-14 13:58:39 +08:00
sre-ci-robot 762d1e8681
Update knowhere commit (#24779)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-09 17:28:36 +08:00
sre-ci-robot bc2112354d
Update knowhere commit (#24742)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-08 14:20:35 +08:00
sre-ci-robot affaf61d78
Update knowhere commit (#24706)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-07 19:44:42 +08:00
Cai Yudong 25aeee5fe6
Update knowhere commit (#24655)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-06-05 19:24:33 +08:00
sre-ci-robot 5caa654622
Update knowhere commit (#24552)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-01 13:44:35 +08:00
congqixia aad2e5d961
Fix compile error after knowhere change prometheus header path (#24434)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-26 15:15:26 +08:00
presburger 152acb3a76
fix fmt lib conflict when compile with raft suppport (#24266)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-05-24 13:05:26 +08:00
sre-ci-robot 7f96dd8526
Update knowhere commit (#24323)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-23 16:15:25 +08:00
sre-ci-robot 785fe4d543
Update knowhere commit (#24288)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-22 15:29:26 +08:00
Cai Yudong b1afd3ea2f
Update knowhere commit to fix BIN_IVF_FLAT upgrade (#24187)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-05-18 17:57:23 +08:00
Cai Yudong 58192f55f6
Update knowhere version for new GetVectorByIds interface (#24034)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-05-11 18:05:20 +08:00
sre-ci-robot b692f90be9
Update knowhere commit (#24032)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-11 15:41:21 +08:00
sre-ci-robot a86332b2d7
Update knowhere commit (#23941)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-08 14:02:40 +08:00
sre-ci-robot 8ffa8a28fa
Update knowhere commit (#23874)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-06 14:18:39 +08:00
Cai Yudong 028054ee0b
Update knowhere version (#23769)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-04-29 10:06:38 +08:00
sre-ci-robot e988256467
Update knowhere commit (#23757)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-04-27 14:44:36 +08:00
sre-ci-robot 8d3007b479
Update knowhere commit (#23727)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-04-27 09:20:43 +08:00
yah01 60fdd7e4f4
Introduce simdjson (#23644)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-26 10:30:34 +08:00
foxspy 6f4ed517de
add growing segment index (#23615)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
sre-ci-robot ccba1334dd
Update knowhere commit (#23595)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-04-21 15:34:31 +08:00
Cai Yudong 2725d38b9e
Add COSINE metric type (#23350)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-04-20 10:20:31 +08:00
Bennu f63d52944f
Update knowhere version from main to commit_id (#23437)
Signed-off-by: Bennu-Li <yunmei.li@zilliz.com>
2023-04-17 16:32:33 +08:00
Enwei Jiao ef57ba3bac
Remove easylogging and replace by GLog (#22979)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-27 18:22:02 +08:00
presburger 5841a3707b
suppport for raft gpu (#22790)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-03-17 11:59:54 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-03-14 23:21:56 +08:00
Enwei Jiao b25b3ef431
Integreation with Velox (#22102)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-16 17:26:35 +08:00
presburger 9950cacd10
support knowhere 2.0 (#21857)
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2023-02-10 14:24:32 +08:00
liliu-z 915503de0b
Upgrade knowhere to 1.3.8 (#22087)
Signed-off-by: Li Liu <li.liu@zilliz.com>
2023-02-09 18:58:32 +08:00
Cai Yudong fc1d2d18cf
Optimize knowhere CMakeLists.txt (#22000)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2023-02-07 16:12:04 +08:00
jaime e05ac56283
Revert build arrow with Conan (#21258)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-16 18:27:24 +08:00
liliu-z 6188960e85
Upgrade Knowhere to 1.3.6 to fix bugs (#21240)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-15 12:15:22 +08:00
liliu-z d326c385bb
Upgrade Knowhere to 1.3.5 to fix DiskANN aio issue (#21109)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-09 22:53:20 +08:00
liliu-z aecf2e4f4b
Upgrade knowhere to 1.3.4 (#20914)
Signed-off-by: Li Liu <li.liu@zilliz.com>

Signed-off-by: Li Liu <li.liu@zilliz.com>
2022-12-02 00:33:21 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-23 10:39:11 +08:00
liliu-z 0dc659a04c
Upgrade Knowhere to 1.3.3 to fix potential bugs (#20434)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-11-09 20:19:04 +08:00
Jiquan Long 377cd210c2
Make knowhere use segcore's cmake build type (#20041)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-10-25 14:17:37 +08:00
zhagnlu 05bbb12d30
Skip build aws sdk when not need diskann (#19965) (#19966)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-10-21 18:35:29 +08:00
zhagnlu 02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854)
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-10-20 10:23:27 +08:00
mumon b48c37cb44
Fix typo in core (#19661)
Signed-off-by: Ziyu Wang <15871035978@163.com>

Signed-off-by: Ziyu Wang <15871035978@163.com>
2022-10-12 17:05:23 +08:00
liliu-z faa9d2176a
Upgrade Knowhere to 1.3.2 to support large dim in DiskANN (#19531)
Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>

Signed-off-by: Li Liu <liliu@LideMacBook-Pro.local>
Co-authored-by: Li Liu <liliu@LideMacBook-Pro.local>
2022-09-29 19:40:55 +08:00
SimFG 68a257458b
Implement the `hook` interceptor (#19294)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-23 09:50:52 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
Jenny Li d8b11121a8
Decrease build time by removing compiling openblas (#19207)
Signed-off-by: Jenny Li <jing.li@zilliz.com>

Signed-off-by: Jenny Li <jing.li@zilliz.com>
2022-09-16 17:46:47 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935)
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-05 10:41:11 +08:00
Cai Yudong 94ffa8a275
Upgrade to knowhere-v1.2.0 (#18746)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-23 10:20:58 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-20 22:22:31 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-07-12 11:58:25 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
Enwei Jiao ea901393ef
fix windows ldflags (#17807)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-28 19:58:25 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-24 10:34:18 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-17 17:46:10 +08:00
bigsheeper cdcdfa1ea5
Disable jemalloc and use malloc_trim instead (#17538)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-14 11:12:09 +08:00
bigsheeper 2d9a52206d
Use jemalloc in QueryNode, DataNode and IndexNode (#17470)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-10 11:24:08 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-02 10:48:03 +08:00
Xiaofan 9579a645c6
Support compile marisa on Macos (#17261)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-31 13:28:02 +08:00
Enwei Jiao d28a2db46c
move arrow from storage to core (#17061)
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-22 20:03:58 +08:00
Cai Yudong 35e7ed45b3
Upgrade to knowhere v1.1.5 to fully support MinGW build (#16498)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-04-15 09:53:34 +08:00
zhenshan.cao 58ea38142f
Use boost dynamic_bitset in segcore (#16476)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-14 22:37:34 +08:00
Ji Bin a5ac637ba5
Adaptation windows for Knowhere 1.1.3 (#16444)
fixes #16443

Signed-off-by: Ji Bin <matrixji@live.com>
2022-04-13 14:39:34 +08:00
Cai Yudong 71cd7ba67a
Add configuration common.indexSliceSize (#16438)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-04-08 20:29:33 +08:00
Cai Yudong a37479d728
Upgrade to knowhere-v1.1.2 to support all index types for mac (#16416)
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2022-04-08 15:29:31 +08:00
Jiquan Long fd589baca7
Integrates marisa trie index (#16192)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-04-01 15:31:29 +08:00
Cai Yudong 0baa8c60a5
Fix MinGW build error (#16267)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-30 00:15:27 +08:00
Cai Yudong 1160693e4c
Fix centos build error (#16251)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-29 15:13:30 +08:00
Cai Yudong 2ac53c89e9
Revert #16242 (#16247)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-29 00:53:27 +08:00
Cai Yudong 9a6b2b8806
Fix centos build error (#16242)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-28 23:23:27 +08:00
Ji Bin 4f08f6d8c6
Workaround for knowhere under windows (#16214)
issue #16205

Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-28 17:41:27 +08:00
Cai Yudong 3e3756d3dc
Upgrade to knowhere v1.1.1 to support Mac M1 build (#16210)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-28 14:33:26 +08:00
Cai Yudong f4ebd3a9ce
Upgrade to knowhere v1.1.0 (#16186)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-25 13:49:25 +08:00
Ji Bin 3cd28420f1
Support compile under windows (#15786)
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
2022-03-17 17:17:22 +08:00
Cai Yudong d65cf5309f
Remove useless LogMgr (#16080)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-16 21:19:21 +08:00
Cai Yudong 529098fd87
[skip ci] Update knowhere/CMakeLists.txt (#15804)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-01 17:55:55 +08:00
Cai Yudong adca79facc
Fix centos build error (#15760)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-25 18:49:54 +08:00
Nemo dc65b1aeea
Switch to knowhere v1.0.1. Also switch to forked gorocksdb v0.0.1 (#15603)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-17 19:05:49 +08:00
jaime 307a8ce535
Support compile and run on Mac (#15491)
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
2022-02-09 14:27:46 +08:00
Ji Bin d83f69bb21
Support windows compliation (#15448)
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
2022-02-07 23:43:46 +08:00
Xieql bdbeac2204
[skip e2e] Improve annotation (#14562)
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
2021-12-30 14:15:38 +08:00
Cai Yudong 1ae249adb5
Update profiler CMakeLists.txt (#13001)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-08 23:23:06 +08:00
Cai Yudong 85efcd8582
Move fiu nlohmann and easylogging to core/thirdparty (#12981)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-08 18:45:07 +08:00
Cai Yudong 45bac3e4ec
Move profiler under core/thirdparty (#12949)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-08 13:01:04 +08:00
Cai Yudong df875c5985
Clean useless thirdparty version (#12944)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-08 12:15:03 +08:00
Cai Yudong 7b97b155e8
Remove duplicated thirdparty (#12925)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-08 10:37:37 +08:00
XuanYang-cn ea7abf49c3
Fix protobuf compilation error (#9881)
resolves:#9876

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-14 18:14:32 +08:00
ZhiShen 6a770e5c38
Make knowhere compile independently (#7606)
Make knowhere compile independently

1. Make knowhere compile independently
    * Add gtest, arrow, and some other libraries to index.
    * Add cache, log and some other files to knowhere.
    * Add CMakeLists files to index's thirdparty.

2. Modified the compilation content of knowhere
    * Delete some content of compile library.
    * Add IMPORTED_GLOBAL property to faiss.

3.  Change the compilation location of some libraries
    * Make OpenBLas compiled in thirdpartycore.cmake.
    * Make faiss compiled in thirdparty/CMakeLists.

Change the content of knowhere/CMakeLists

1. Change easyloggingpp and nlohmann into index/thirdparty.
2. Change MILVUS_THIRDPARTY_SRC into KNOWHERE_THIRDPARTY_SRC.

Delete FindOpenBLAS

1. Delete Openblas.cmake.

2. The search task for openBlas is assigned to ThirdpartyCore.

3. Some changes were made to build.sh in index.

Fix the openBLas compilation problem

Delete the if-else in compilation of faiss;

Now when complie faiss, it will find the Openblas as we wish.

Fix some problem:

1. delete arrow

2. set openblas_source to AUTO

3. change a include_dir

4. delete MKL

5. delete the CMakeLists in index/utils,cache,log

Change variable build_test to knowhere_build_test in index/build.sh

Change the include location of  GNUInstallDirs

set CMAKE_INSTALL_LIBDIR

Resolves: milvus-io#5183
See also: milvus-io#6604

Signed-off-by: Shen Zhi <m13120163046@163.com>
2021-10-13 17:06:33 +08:00
quicksilver 3db653f2d0
Update cpp thirdparty source URL (#5295)
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

Resolves: #5293
2021-05-19 09:39:29 +00:00
XuanYang-cn 46d27e37d7 Skip install gtest when build with test
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-08 10:01:18 +08:00
FluorineDog af1900b42a Remove ConcurrentBitsetPtr in segcore
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-03-09 16:16:43 +08:00
FluorineDog ef98dab2a9 Support segcore config
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-03-04 17:09:48 +08:00
FluorineDog 66146223ca Support flat
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-02-27 12:46:37 +08:00
xige-16 4c491471ee Add release collection and release partition interface for query node
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-24 15:58:55 +08:00
xige-16 7a7a73e89c Fix high memory usage in pulsarTtStream
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-23 11:40:12 +08:00
FluorineDog 15dd17488e Support benchmark
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-02-23 10:47:21 +08:00
GuoRentong 1104f059ee Update doc:module interfaces
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-13 11:08:03 +08:00
quicksilver 8e9d8e36e1 Update run_go_unittest.sh
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-13 10:40:46 +08:00
FluorineDog 4cd42c553f Rename field_name, make field_id strongly typed, skip multithread test
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-12 18:31:52 +08:00