Commit Graph

1325 Commits (cbba262442a42cc22ae24e00842d901b58321c97)

Author SHA1 Message Date
yukun cbba262442
Fix metric_type bug in search (#3155)
* Fix dsl test case nb bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix dsl test case bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add metric_type judge in search

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_search.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_db

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix search metric_type

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* ci retry

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-06 19:08:42 +08:00
XuanYang-cn 38a3fe766d
Move gtest download and build process to thirdparty/gtest (#3145)
* fix cmake configure warning

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add config tests to ctest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* rm customed add_test, using default add_test

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add thirdparty/gtest and built-able

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add dependences fiu

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add license, rm not useful comments

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix include dir not exist bug

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add some config

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* gtest download at configure time

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-06 19:07:20 +08:00
groot d22ff25a35
fix delete eneity bug (#3161)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-08-06 18:18:25 +08:00
groot c73a58a114
add unittest case FetchTest (#3153)
* add unittest cases

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* add unittest case

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-06 14:07:11 +08:00
op-hunter 7688f51343
Faiss hnsw upgrade (#3134)
* combine the hnsw's implemention of faiss and hnswlib

Signed-off-by: cmli <chengming.li@zilliz.com>

* transplant the datastructure of hnsw from hnswlib 2 faiss

Signed-off-by: cmli <chengming.li@zilliz.com>

* basic work finished, pass compile, to be tested

Signed-off-by: cmli <chengming.li@zilliz.com>

* rhnswflat, rhnswsq, rhnswpq pass ut

Signed-off-by: cmli <chengming.li@zilliz.com>

* remove AssertAnns of RHNSWPQ because PQ has accuracy loss

Signed-off-by: cmli <chengming.li@zilliz.com>


Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
2020-08-06 11:36:41 +08:00
yukun ef24a8b340
Fix dsl test cases bug (#3130)
* Fix dsl test case nb bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix dsl test case bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add metric_type judge in search

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_search.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_db

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-08-05 21:44:17 +08:00
groot 2b2ebc27e5
fix auto-generated id field bug (#3137)
* autogen id

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix bug

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-05 18:19:51 +08:00
BossZou fc393b3aeb
(scalar) Debug delete request (#3107)
* Debug delete request

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Chang previous bloom filter and deleted docs file to stale during ApplyDeletes

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Add delete case

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Add delete ut

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Add Delete test

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Format

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-05 16:27:58 +08:00
chen qingxiang aad8f82d41
fix index bug caused by json parse (#3129)
* fix index bug caused by json parse

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2020-08-05 16:24:47 +08:00
groot 3dfbe61422
modify proto (#3126)
* modify proto

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix compact

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* print cache

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-05 14:02:51 +08:00
groot 362bf7a94e
cache (#3124)
* cache

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-05 10:04:41 +08:00
Wang Xiangyu f18c95eed3
update grpc commit (#3125)
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-05 09:44:10 +08:00
yukun b12a19df34
Fix bug (#3121)
* Fix Block Format Read bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix Search crash bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix CreateCollection bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add db->Query unittest

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index name bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_db error

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-04 18:29:50 +08:00
Wang Xiangyu 7861da3e92
[skip ci]update grpc commit and md5 (#3123)
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-04 18:22:33 +08:00
BossZou 9aa364ce25
Optimize DeletedDocsFormat class file I/O (#3112)
* Optimize DeletedDocsFormat file I/O

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Update DeletedDocsFormat Read

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-04 15:45:44 +08:00
yukun 96d799655a
Add DBImpl::Query unittest (#3119)
* Fix Block Format Read bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix Search crash bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix CreateCollection bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add db->Query unittest

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-04 15:00:15 +08:00
groot 1f705a2096
define binary data type (#3115)
* define binary data type

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
2020-08-04 14:58:18 +08:00
Xiaohai Xu 19c5dc079f
add logging (#3092)
Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
2020-08-04 12:09:15 +08:00
groot 8f8fa0aaf2
use index type replace name (#3114)
* use index type replace name

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-08-04 10:45:41 +08:00
XuanYang-cn 6ecf7a276e
fix segment dependency sqlite (#3113)
* fix segment dependency

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add dependence mysqlpp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* quick fix

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
2020-08-03 18:07:01 +08:00
chen qingxiang 62841c68d5
fix bug caused by wrong valid row initlization (#3108)
* fix bug caused by wrong valid row initlization

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* fix bug caused by wrong valid row initlization

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

Co-authored-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-03 15:47:48 +08:00
Wang Xiangyu 1232011de7
[skip ci]copy dablooms' license file from origin repository (#3110)
Signed-off-by: Wang Xiangyu <xy.wang@zilliz.com>
2020-08-03 15:44:40 +08:00
yukun 7e1fd3d1e8
Fix bug (#3106)
* Fix Block Format Read bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix Search crash bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix CreateCollection bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-03 11:59:02 +08:00
XuanYang-cn 28af7eb4a7
Rearrange db/CMKLists, seperate dir codecs, storage, cache and segment (#3105)
* change cmake minmum version requires to 3.14

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart server/CMKLists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart cache

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart storage

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart codecs

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart segment

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* rm thirdparty dablooms from db/cmklists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add comments about thirdparty lib

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* add license

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-03 11:58:35 +08:00
BossZou f1d271c0e9
(db/snapshot) Optimize mysql (#3095)
* Move MysqlPool to backend

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Update header

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Optimize mysqlpool

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-03 11:58:08 +08:00
groot fd2e3de65d
fix build error (#3104)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-03 11:20:44 +08:00
yukun 1c54baeec3
Fix bug (#3099)
* Fix Block Format Read bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix Search crash bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-08-03 10:41:48 +08:00
XuPeng-SH dbff431d2a
(db/snapshot): Add new TypeNameField and update some ut (#3100)
* (db/snapshot): Add TypeNameField

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add test for DropAllIndexOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

Co-authored-by: Jin Hai <hai.jin@zilliz.com>
2020-08-03 09:54:55 +08:00
BossZou 69f4f4d54b
(db/snapshot) implement Delete (#3101)
* Update API DeleteEntity in MemMgr

Signed-off-by: yhz <413554850@qq.com>

* Stop flush automaticlly

Signed-off-by: yhz <413554850@qq.com>

* Doing ApplyDeletes

Signed-off-by: yhz <413554850@qq.com>

* Mysql Schema formt

Signed-off-by: yhz <413554850@qq.com>

* Delete code done

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Delete run pass

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>

* Allow auto flush

Signed-off-by: yinghao.zou <yinghao.zou@zilliz.com>
2020-08-03 09:51:37 +08:00
chen qingxiang 3f02104265
fix bugs when GetEntity with ID in mulity paritions (#3098)
* fix bugs when GetEntity with ID in mulity paritions

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* cancel annotation of index thread

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2020-08-01 18:32:55 +08:00
Cai Yudong 4024e5cc03
fix ci (#3097)
* rename CollectionMappings to FieldElementMappings

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-08-01 17:03:05 +08:00
Wang XiangYu 9704db4c40
fix compile error (#3094)
Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-08-01 12:02:38 +08:00
XuPeng-SH 4e143775de
(db/snapshot): Update DropAllIndexOperation and some small changes (#3087)
* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Segment File Operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): crtp for BaseResource

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add InActiveResourcesGCEvent

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix ut error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update snapshot segmentcommit operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update drop all index operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix gc

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix gc segment files

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): refactor Event related

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change 1

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change for GC event

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix build error for high version of boost filesystem

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update compound operations

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add operation test

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for CompoundSegmentsOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add test for CompoundSegmentsOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add time stat in operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling 2

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling 3

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small refactor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for DropAllIndexOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-01 11:06:57 +08:00
XuanYang-cn a85660bef8
Apart src/CMKLists to dir server/CMKLists (#3081)
* change cmake minmum version requires to 3.14

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart server/CMKLists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-01 10:55:40 +08:00
yukun d0b3c74f9a
Fix Block Format Read bug (#3084)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-08-01 10:55:17 +08:00
groot 5c37a9faa3
refine insert code (#3090)
* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine insert code

Signed-off-by: groot <yihua.mo@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-01 10:17:32 +08:00
chen qingxiang 9d464bde1d
fix get entity by id bug (#3086)
* fix get entity by id bug

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* cancel annotation of index thread

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* format code

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-01 09:45:51 +08:00
groot f2a22a09e3
refine code (#3089)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-08-01 09:10:46 +08:00
groot 36a1dea3b1
describe index request (#3085)
* describe index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build error

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-31 17:48:57 +08:00
Cai Yudong e426b18d0e
fix ci (#3080)
* rename partition name to tag

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move BaseReq/ReqType to Types.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt BaseReq

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-07-31 15:48:19 +08:00
groot d0543d21e3
fix bug (#3078)
* fix bug

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-31 14:21:06 +08:00
yukun feb29256dc
Fix GetEntityByID bug (#3073)
* Add search implementation in ExecutionEngine

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove attr_type from query_ptr

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add CreateStructuredIndex in ExecutionEngine

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add index fields in QueryPtr

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix SearchReq bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* code format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add Search in scheduler

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix SearchJob and SearchTask

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add MergeTopkResultSet

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove nq in VectorQuery

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change segment_size to segment_row_count in C++ sdk

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix row_count in Search

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-07-31 11:44:15 +08:00
BossZou 007356af41
(scalar) Update list-id-in-segment case (#3060)
* Update test list id

Signed-off-by: yhz <413554850@qq.com>

* Update list id in segment case

Signed-off-by: yhz <413554850@qq.com>
2020-07-31 10:33:31 +08:00
groot 2c3fc9684a
remove MetaTypes.h (#3072)
* remove MetaTypes.h

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* remove MetaConsts.h

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-31 09:42:20 +08:00
XuPeng-SH 1449cf0932
(db/snapshot): Handle timeout with external metastore (#3075)
* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Segment File Operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): crtp for BaseResource

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add InActiveResourcesGCEvent

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix ut error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update snapshot segmentcommit operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update drop all index operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix gc

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix gc segment files

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): refactor Event related

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change 1

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change for GC event

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix build error for high version of boost filesystem

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update compound operations

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add operation test

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for CompoundSegmentsOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add test for CompoundSegmentsOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add time stat in operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling 2

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): timeout handling 3

Signed-off-by: peng.xu <peng.xu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-07-30 22:31:54 +08:00
Cai Yudong 68e1a8da2b
add field name and segment size validation check when create collection (#3065)
* add field name and segment size validation check when create collection

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add collection name check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* clean header files

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move constants to Types.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-07-30 18:17:10 +08:00
yukun 5e59056acf
Fix DBImpl and scheduler for search (#3062)
* Add search implementation in ExecutionEngine

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove attr_type from query_ptr

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add CreateStructuredIndex in ExecutionEngine

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add index fields in QueryPtr

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix SearchReq bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* code format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add Search in scheduler

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix SearchJob and SearchTask

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add MergeTopkResultSet

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove nq in VectorQuery

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change segment_size to segment_row_count in C++ sdk

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix row_count in Search

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-07-30 17:07:14 +08:00
XuanYang-cn 7cc4862d4f
[skip ci]quick fix (#3071)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
2020-07-30 17:06:20 +08:00
groot 841a7586d9
get collection stats (#3069)
* build index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix some problem

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build index ut

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* avoid ut error

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* get collection stats

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* remove EngineType

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-30 16:55:51 +08:00
XuanYang-cn 86652416c5
ThirdParty libs won't be re-downloaded after make clean (#3059)
* reformate all CMKLists to make it more readable

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Thirdparty libs won't be re-download after make clean

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-07-30 15:14:16 +08:00