Commit Graph

57 Commits (cbba262442a42cc22ae24e00842d901b58321c97)

Author SHA1 Message Date
groot fd2e3de65d
fix build error (#3104)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-03 11:20:44 +08:00
XuanYang-cn a85660bef8
Apart src/CMKLists to dir server/CMKLists (#3081)
* change cmake minmum version requires to 3.14

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* apart server/CMKLists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-08-01 10:55:40 +08:00
XuanYang-cn 86652416c5
ThirdParty libs won't be re-downloaded after make clean (#3059)
* reformate all CMKLists to make it more readable

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Thirdparty libs won't be re-download after make clean

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Wang XiangYu <xy.wang@zilliz.com>
2020-07-30 15:14:16 +08:00
XuanYang-cn 4b52c573fb
Remove repeated built src files in unittest/CMakeLists (#3047)
* add LICENSE to CMKLists.txt

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Rearrange utils CMKLists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* mov main.cpp to to CMKLists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* tidy src/db CMKLists.txt

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove repeated build src files in unittest/CMKLists and /unittest/db/CMKlists

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix format

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix typo

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
2020-07-28 17:26:20 +08:00
groot 34309709ed
refine scheduler job task (#3044)
* remove SS

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* modify pr template

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine job task

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-28 14:12:25 +08:00
groot 43cfe506ac
remove unused code (#3024)
Signed-off-by: groot <yihua.mo@zilliz.com>
2020-07-27 11:38:45 +08:00
groot 8b3b15c980
refine code (#3022)
* clean code first stage

Signed-off-by: groot <yihua.mo@zilliz.com>

* rename some files

Signed-off-by: groot <yihua.mo@zilliz.com>

* rename

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-07-26 17:36:06 +08:00
Wang XiangYu dfdb5cfbe4
log module (#3017)
Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-07-25 19:46:34 +08:00
Wang XiangYu 7057f13e66
fix unittest test_ssdb failed (#3013)
Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-07-25 17:01:11 +08:00
groot 67809c6e2e
build snapshot index (#2979)
* implement build index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix warning

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* compress missed

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* ssdb

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-07-23 15:53:11 +08:00
Wang XiangYu 4288097aa0
Rewrite config module (#2962)
* rewrite config module

Signed-off-by: wxyu <xy.wang@zilliz.com>

* clang-format

Signed-off-by: wxyu <xy.wang@zilliz.com>

* remove empty test

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix include

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix compile

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix clang-format

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-07-23 15:27:07 +08:00
yukun fd4969edaf
Merge scalar-field-filtering to master (#2976)
* [skip ci] remove timeout in partition-test case

Signed-off-by: zw <zw@milvus.io>

* Update server_versiong

Signed-off-by: zw <zw@milvus.io>

* fix client_test.go

Signed-off-by: zw <zw@milvus.io>

* Add ci param is_manual_trigger

Signed-off-by: zw <zw@milvus.io>

* update ci param

Signed-off-by: zw <zw@milvus.io>

* Replace wal to hybrid

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Reconstruct create structured index

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* add filter cases

Signed-off-by: zw <zw@milvus.io>

* Change interfaces for hybrid search in GrpcHandler

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc for new proto

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Filter case (#2655)

* [skip ci] remove timeout in partition-test case

Signed-off-by: zw <zw@milvus.io>

* Update server_versiong

Signed-off-by: zw <zw@milvus.io>

* fix client_test.go

Signed-off-by: zw <zw@milvus.io>

* Add ci param is_manual_trigger

Signed-off-by: zw <zw@milvus.io>

* update ci param

Signed-off-by: zw <zw@milvus.io>

* add filter cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix all compile errors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Comment all test_db

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* undo clange format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci]Filter case (#2688)

* [skip ci] remove timeout in partition-test case

Signed-off-by: zw <zw@milvus.io>

* Update server_versiong

Signed-off-by: zw <zw@milvus.io>

* fix client_test.go

Signed-off-by: zw <zw@milvus.io>

* Add ci param is_manual_trigger

Signed-off-by: zw <zw@milvus.io>

* update ci param

Signed-off-by: zw <zw@milvus.io>

* add filter cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix GetEntityByID bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] fix some case bug

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update some cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update some cases (#2700)

* [skip ci] remove timeout in partition-test case

Signed-off-by: zw <zw@milvus.io>

* Update server_versiong

Signed-off-by: zw <zw@milvus.io>

* fix client_test.go

Signed-off-by: zw <zw@milvus.io>

* Add ci param is_manual_trigger

Signed-off-by: zw <zw@milvus.io>

* update ci param

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update some cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Using json to pass TermQuery and RangeQuery

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix C++ sdk

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix C++ sdk compile error

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* GetEntityByID returns all entities when no field names provide

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add changelog

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Comment test_db and ssdb

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Collection create success if an invalid field name provided

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* The number of fields should be limited

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Check the validity of the parameters of creating collection: segment_size

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Index can not be created

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* add changelog

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Count entities got wrong result with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* No entity returned with get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Index type name should returned if index type is not supported

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* The max value of top-k should be limited

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server destroyed after delete by id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* code format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* code format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix C++ sdk

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed when calling get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove test_case change

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* Unexpected error when insert binary entities

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add field_names judge in GetEntityByID

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned if searching with tags

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] update search cases (#2795)

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Filter case (#2825)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Distances returned by calling search is inaccurate (#2820)

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id (#2840)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Filter case (#2848)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Test scalar field filtering (#2861)

* fix test_flush.py to meet the entity of branch scalar_field_filtering

* update test_flush.py to meet hybrid search

* Filter case (#2876)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Server crashed after calling delete_entity_by_id (#2870)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Test scalar field filtering (#2881)

* fix test_flush.py to meet the entity of branch scalar_field_filtering

* update test_flush.py to meet hybrid search

* fix test_flush.py to meet the entity of branch scalar_field_filtering

* update test_flush.py to meet hybrid search

* update test_compact.py for hybrid query

* Filter case (#2898)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] update index params gen  (#2911)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] fix cases (#2913)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] Filter case (#2914)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] Filter case (#2915)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] Filter case (#2916)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] Filter case (#2919)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* [skip ci] Filter case (#2922)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Create index failed with binary vectors (#2895)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Filter case (#2928)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update list id in segment

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix GET_TIMEOUT in test_list_id_in_segment.py (#2929)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_list_id_in_segment.py bugs (#2935)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_list_id_in_segment.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Filter case (#2938)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update list id in segment

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update casses

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix test_search.py bugs (#2951)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_list_id_in_segment.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Filter case (#2956)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update list id in segment

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update casses

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix test_search.py invalid_param bugs (#2959)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_list_id_in_segment.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py invalid_param bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_partition.py bugs (#2963)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_list_id_in_segment.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py invalid_param bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_index.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_partition.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Filter case (#2964)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update list id in segment

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update casses

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update level2

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Filter case (#2966)

* [skip ci] update search cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update cases

Signed-off-by: zw <zw@milvus.io>

* update cases

Signed-off-by: zw <zw@milvus.io>

* update index string

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index params gen

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update sq8h to cpu_not_support

Signed-off-by: zw <zw@milvus.io>

* [skip ci] assert entity is None entity not in collection

Signed-off-by: zw <zw@milvus.io>

* [skip ci] todo: check segment_size 1.0

Signed-off-by: zw <zw@milvus.io>

* [skip ci] change field name in cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update list id in segment

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update casses

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update index cases

Signed-off-by: zw <zw@milvus.io>

* [skip ci] update level2

Signed-off-by: zw <zw@milvus.io>

* [skip ci] fix insert cases

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Fix test_compact.py (#2967)

* Distances returned by calling search is inaccurate

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Wrong result returned by get_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed during inserting, and can not restart

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Server crashed after calling delete_entity_by_id

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetEntityByID valid_row

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix index type name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Create index failed with binary vectors

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix GetCollectionInfo interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add extra_params in DescribeCollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Insert binary data failed

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix binary index issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix wrong field name problem

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc.cpp

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix assert status.OK() bugs in test cases

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci]Fix GET_TIMEOUT in test_list_id_in_segment.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_list_id_in_segment.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_search.py invalid_param bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_index.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_partition.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_insert.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_insert.py bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang-format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* [skip ci] Fix test_compact.py

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix C++ sdk DataType

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix collection test cases bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: zw <zw@milvus.io>
Co-authored-by: wxyu <xy.wang@zilliz.com>
Co-authored-by: del-zhenwu <56623710+del-zhenwu@users.noreply.github.com>
Co-authored-by: ThreadDao <zongyufen@foxmail.com>
2020-07-23 11:21:34 +08:00
BossZou a8033a5954
(db/snapshot) Add snapshot DB module (#2851)
* New db impl

Signed-off-by: yhz <413554850@qq.com>

* Add db impl for snapshot

Signed-off-by: yhz <413554850@qq.com>

* Update operations step holders

Signed-off-by: yhz <413554850@qq.com>

* Update snapshot dbimpl & create collection test pass

Signed-off-by: yhz <413554850@qq.com>

* Delete operation pass

Signed-off-by: yhz <413554850@qq.com>

* Add constraints for db table

Signed-off-by: yhz <413554850@qq.com>

* Update index operation steps behavior

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add lock in mysql engine and remove lock in store

Signed-off-by: yhz <413554850@qq.com>

* Index unittest case pass

Signed-off-by: yhz <413554850@qq.com>

* Add merge operation step context

Signed-off-by: yhz <413554850@qq.com>

* Remove constraints in DB and snapshot ut pass

Signed-off-by: yhz <413554850@qq.com>

* update DBImple

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add DBHelper to generate sql & add mock Mem db & snaoshot ut pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add mock db engine

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Update DB select API using template for resource

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Update snapshot db name and move them under folder meta

Signed-off-by: yhz <413554850@qq.com>

* clang-format

Signed-off-by: yhz <413554850@qq.com>
2020-07-14 19:11:24 +08:00
Cai Yudong 2c3b3a5cfa
snapshot integrate (#2814)
* update HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add IterateHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* IterateHandler opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentReader

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add src/codecs/snapshot

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update latest code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentReader interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentWriter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentWriter interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest/db build fail

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use SSSegmentReader and SSSegmentWriter in SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-07-11 13:26:31 +00:00
Cai Yudong cfa5b14e73
add SSDBImpl and ssdb for ut (#2683)
* rename StatusField to StateField

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix typo

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable GC in OnNoRefCallBack

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSDBimpl and unittest/ssdb

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-06-29 10:08:25 +08:00
Cai Yudong 4ce1e6b2fc
optimize unittest build (#2589)
* optimize test_storage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize test_metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize test_server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optmize test_server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize test_schedule

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move test_rpc and test_web to db

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-06-17 17:22:54 +08:00
Cai Yudong a9df83766a
clean util dependency (#2576)
* move part of APIs from utils/ValidationUtil to server/ValidationUtils

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move part of APIs from utils/ValidationUtil to config/Utils

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move part of APIs from utils/CommonUtil to config/Utils

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move EraseFromCache from utils/CommonUtil to db/Utils

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove SignalHandler dependency with server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update fiu

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* retry ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-06-16 22:13:25 +08:00
XuPeng-SH 96166e1a5b
Metadata Snapshot Implementation Phase I (#2414)
* (db/snapshot): migrate code from milvus_metatdata_design repo

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add license info

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add namespace snapshot

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add test_snapshot unit test

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add ScopedResourceTest

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): refactor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add ResourceHodlersTest

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update ResourceHodlersTest

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Operation Test and fix some bug

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Update Operation Test

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Update Operation Test to test build

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Reset API for static instance

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix snapshot bug

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix dead lock bug

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix bug in concurrent mode

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in snapshot 1

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in snapshot 2

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in snapshot 3

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in snapshot 4

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix lint error in snapshot 5

Signed-off-by: peng.xu <peng.xu@zilliz.com>
2020-05-25 10:50:34 +08:00
groot f32cba6111
Merge strategy (#2277)
* merge manager

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix build error

Signed-off-by: groot <yihua.mo@zilliz.com>

* layered merge strategy

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

* use simple

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix parallel multi collections query crashissue

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-05-11 01:38:50 +00:00
BossZou 1811254de6
Obtain running rpc requests information (#2240) (#2241)
* add cmd to get requests

Signed-off-by: yhz <413554850@qq.com>

* forward class declaration

Signed-off-by: yhz <413554850@qq.com>

* Obtain running rpc requests information (fix #2240)

Signed-off-by: yhz <413554850@qq.com>

* Fix unittest compile failed issue

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Log error with msg

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-05-08 11:17:48 +00:00
BossZou 8edb7b6a09
add cpu instruction sets check (#2055)
Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-26 00:59:25 +08:00
Wang XiangYu c0d5a47eba
rename optimizer to selector (#2067)
* rename

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix clang-format

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-24 11:42:17 +00:00
yukun 3e1b2ab4a0
Filtering by numeric scalar fields prototype (#1919)
* Add hybrid request handler

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add C++ sdk for createcollection and insertentities

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add RequestHandler

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add test case for hybrid insert

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix sqlite bug for createcollection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add HybridQuery Handler DBImpl and ExecBinaryQuery

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add HybridSearch sdk

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add HybridSearch test case

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix HybridSearch bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix HybridSearch crash bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change void* to vector<uint8_t> in Attr codec

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add context and new search task

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add merge for Hybrid

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add AST validation

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add unittest for hybrid

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix hybrid search nq bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix bugs after merge master

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix unittest bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix Codacy

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix compact unittest bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Remove grpc request in hybridsearchcontext

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix some codacy quality issue

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix HYBRID_DB_TEST bug

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Annotate new search task

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add Hybrid RPC handler unittest

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix logs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add HybridSearch unittest

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-04-16 14:54:12 +08:00
BossZou 354f29a058
Refactor log on search and insert request (#1887)
* fix stash conflict

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Log config information and device information (fix #1893)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add index annoy into http module (fix #1882)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Refactor log on search and insert request (fix #1886)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* use vsnprintf to avoid oom

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix PR Quality problem & remove surplus comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* code format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* replace strlen with strnlen & remove some surplus comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-09 21:04:18 +08:00
Cai Yudong 66e38d20cd
Caiyd 1881 bad alloc (#1892)
* #1881 update storage APIs, use int64_t for size

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-04-09 17:13:15 +08:00
Cai Yudong dc707bb8a3
Caiyd 1858 disable s3 build (#1860)
* #1858 disable s3 build

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable s3 in unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-04-03 16:35:48 +08:00
groot 99470b4dfa
#1702 Optimize request handler to combine similar query (#1727)
* #1686 API search_in_files cannot work correctly when vectors is stored in certain non-default partition

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* #1686 API search_in_files cannot work correctly when vectors is stored in certain non-default partition

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine request code

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>

* combine search request

Signed-off-by: groot <yihua.mo@zilliz.com>

* combine search request

Signed-off-by: groot <yihua.mo@zilliz.com>

* #1728 Optimize request handler to combine similar query

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>

* remove some log
2020-03-23 18:26:18 +08:00
Cai Yudong 588ef95d76
Caiyd refactor knowhere (#1687)
* add new knowhere

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* build pass

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* update

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* update

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* rename algo

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* update...

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* add archive

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* add new APIs: GetVectorById/SearchVectorById/SetBlacklist/GetBlacklist

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* all unittest pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* test_binary pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move knowhere into namespace milvus

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update hnsw

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ConfAdapterMgr

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ExecutionEngineImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move SetBlacklist/GetBlacklist to VecIndex

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update VectorAdapter and rename SearchById to QueryById

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update interface in ExecutionEngineImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* milvus build pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* milvus IDMAP sdk_simple pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix test_server

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix test_schedule

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix CPU version Milvus build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update BinaryIVF BuildAll

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update VecIndexFactory

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ConfAdapter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix knowhere unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix SPTAG unittest

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix CPU version unittest build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix db_test

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix test_engine

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix test_delete

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix CPU version build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change BinarySet key back

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change IndexType to string, and add compatible API for 0.7.0

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix IndexHNSW build warning

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix test_cache

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix write_index error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code clean

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: Nicky <nicky.xj.lin@gmail.com>
Co-authored-by: xiaojun.lin <xiaojun.lin@zilliz.com>
2020-03-19 19:24:20 +08:00
yudong.cai 28a0f1de01 #1548 move store/Directory.cpp to storage
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-03-12 11:38:17 +08:00
yudong.cai 67fad34065 Merge remote-tracking branch 'main/master' into caiyd_1547_rename_storage_file 2020-03-11 18:43:02 +08:00
yudong.cai 1d18c0a938 #1547 rename storage/file to storage/disk
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-03-09 13:08:17 +08:00
yudong.cai 6b81dd55df #1536 move Config.cpp to config directory
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-03-07 17:02:21 +08:00
BossZou df47fad78c
Add setter API for config `preload_table` (#1526)
* update code

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize http module code

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* move gpu res handler to config holder

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add setter API for config preload_table (fix #1525)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add else branch at end of if condition

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-03-06 13:26:01 +08:00
Jin Hai dab74700b2
Delete and WAL feature branch merge (#1436)
* add read/write lock

* change compact to ddl queue

* add api to get vector data

* add flush / merge / compact lock

* add api to get vector data

* add data size for table info

* add db recovery test

* add data_size check

* change file name to uppercase

Signed-off-by: jinhai <hai.jin@zilliz.com>

* update wal flush_merge_compact_mutex_

* update wal flush_merge_compact_mutex_

* change requirement

* change requirement

* upd requirement

* add logging

* add logging

* add logging

* add logging

* add logging

* add logging

* add logging

* add logging

* add logging

* delete part

* add all size checks

* fix bug

* update faiss get_vector_by_id

* add get_vector case

* update get vector by id

* update server

* fix DBImpl

* attempting to fix #1268

* lint

* update unit test

* fix #1259

* issue 1271 fix wal config

* update

* fix cases

Signed-off-by: del.zhenwu <zhenxiang.li@zilliz.com>

* update read / write error message

* update read / write error message

* [skip ci] get vectors by id from raw files instead faiss

* [skip ci] update FilesByType meta

* update

* fix ci error

* update

* lint

* Hide partition_name parameter

* Remove douban pip source

Signed-off-by: zhenwu <zw@zilliz.com>

* Update epsilon value in test cases

Signed-off-by: zhenwu <zw@zilliz.com>

* Add default partition

* Caiyd crud (#1313)

* fix clang format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add faiss_bitset_test

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* avoid user directly operate partition table

* fix has table bug

* Caiyd crud (#1323)

* fix clang format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use compile option -O3

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update faiss_bitset_test.cpp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change open flags

* change OngoingFileChecker to static instance

* mark ongoing files when applying deletes

* update clean up with ttl

* fix centos ci

* update

* lint

* update partition

Signed-off-by: zhenwu <zw@zilliz.com>

* update delete and flush to include partitions

* update

* Update cases

Signed-off-by: zhenwu <zw@zilliz.com>

* Fix test cases crud (#1350)

* fix order

* add wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix invalid operation issue

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix invalid operation issue

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix bug

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix bug

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* crud fix

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* crud fix

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* add table info test cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* merge cases

Signed-off-by: zhenwu <zw@zilliz.com>

* Shengjun (#1349)

* Add GPU sharing solution on native Kubernetes  (#1102)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Fix http server bug (#1096)

* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* change gpu resources invoke way to cmd()

* set advanced config name add DEFAULT

* change config setting to cmd

* modify ..

* optimize code

* assign TableDto' count default value 0 (fix #995)

* check if table exists when show partitions (fix #1028)

* check table exists when drop partition (fix #1029)

* check if partition name is legal (fix #1022)

* modify status code when partition tag is illegal

* update changlog

* add info to /system url

* add binary index and add bin uri & handler method(not completed)

* optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067)

* fix test partition bug

* fix test bug when check insert records

* add binary vectors test

* add default for offset and page_size

* fix uinttest bug

* [skip ci] remove comments

* optimize web code for PR comments

* add new folder named utils

* check offset and pagesize (fix #1082)

* improve error message if offset or page_size is not legal (fix #1075)

* add log into web module

* update changlog

* check gpu sources setting when assign repeated value (fix #990)

* update changlog

* clang-format pass

* add default handler in http handler

* [skip ci] improve error msg when check gpu resources

* change check offset way

* remove func IsIntStr

* add case

* change int32 to int64 when check number str

* add log in we module(doing)

* update test case

* add log in web controller

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>

* Filtering for specific paths in Jenkins CI  (#1107)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Fix Filtering for specific paths in Jenkins CI bug (#1109)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Fix Filtering for specific paths in Jenkins CI bug (#1110)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Don't skip ci when triggered by a time (#1113)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Don't skip ci when triggered by a time

* Don't skip ci when triggered by a time

* Set default sending to Milvus Dev mail group  (#1121)

* run hadolint with reviewdog

* add LINCENSE in Dockerfile

* run hadolint with reviewdog

* Reporter of reviewdog command is "github-pr-check"

* format Dockerfile

* ignore DL3007 in hadolint

* clean up old docker images

* Add GPU sharing solution on native Kubernetes

* nightly test mailer

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Test filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* Filtering for specific paths in Jenkins CI

* No skip ci when triggered by a time

* Don't skip ci when triggered by a time

* Set default sending to Milvus Dev

* Support hnsw (#1131)

* add hnsw

* add config

* format...

* format..

* Remove test.template (#1129)

* Update framework

* remove files

* Remove files

* Remove ann-acc cases && Update java-sdk cases

* change cn to en

* [skip ci] remove doc test

* [skip ci] change cn to en

* Case stability

* Add mail notification when test failed

* Add main notification

* Add main notification

* gen milvus instance from utils

* Distable case with multiprocess

* Add mail notification when nightly test failed

* add milvus handler param

* add http handler

* Remove test.template

Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>

* Add doc for the RESTful API / Update contributor number in Milvus readme (#1100)

* [skip ci] Update contributor number.

* [skip ci] Add RESTful API doc.

* [skip ci] Some updates.

* [skip ci] Change port to 19121.

* [skip ci] Update README.md.

Update the descriptions for OPTIONS.

* Update README.md

Fix a typo.

* #1105 update error message when creating IVFSQ8H index without GPU resources (#1117)

* [skip ci] Update README (#1104)

* remove Nvidia owned files from faiss (#1136)

* #1135 remove Nvidia owned files from faiss

* Revert "#1135 remove Nvidia owned files from faiss"

This reverts commit 3bc007c28c.

* #1135 remove Nvidia API implementation

* #1135 remove Nvidia owned files from faiss

* Update CODE_OF_CONDUCT.md (#1163)

* Improve codecov (#1095)

* Optimize config test. Dir src/config 99% lines covered

* add unittest coverage

* optimize cache&config unittest

* code format

* format

* format code

* fix merge conflict

* cover src/utils unittest

*  '#831 fix exe_path judge error'

* #831 fix exe_path judge error

* add some unittest coverage

* add some unittest coverage

* improve coverage of src/wrapper

* improve src/wrapper coverage

* *test optimize db/meta unittest

* fix bug

* *test optimize mysqlMetaImpl unittest

* *style: format code

* import server& scheduler unittest coverage

* handover next work

* *test: add some test_meta test case

* *format code

* *fix: fix typo

* feat(codecov): improve code coverage for src/db(#872)

* feat(codecov): improve code coverage for src/db/engine(#872)

* feat(codecov): improve code coverage(#872)

* fix config unittest bug

* feat(codecov): improve code coverage core/db/engine(#872)

* feat(codecov): improve code coverage core/knowhere

* feat(codecov): improve code coverage core/knowhere

* feat(codecov): improve code coverage

* feat(codecov): fix cpu test some error

* feat(codecov): improve code coverage

* feat(codecov): rename some fiu

* fix(db/meta): fix switch/case default action

* feat(codecov): improve code coverage(#872)
* fix error caused by merge code
* format code

* feat(codecov): improve code coverage & format code(#872)

* feat(codecov): fix test error(#872)

* feat(codecov): fix unittest test_mem(#872)

* feat(codecov): fix unittest(#872)

* feat(codecov): fix unittest for resource manager(#872)

* feat(codecov): code format (#872)

* feat(codecov): trigger ci(#872)

* fix(RequestScheduler): remove a wrong sleep statement

* test(test_rpc): fix rpc test

* Fix format issue

* Remove unused comments

* Fix unit test error

Co-authored-by: ABNER-1 <ABNER-1@users.noreply.github.com>
Co-authored-by: Jin Hai <hai.jin@zilliz.com>

* Support run dev test with http handler in python SDK (#1116)

* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* change gpu resources invoke way to cmd()

* set advanced config name add DEFAULT

* change config setting to cmd

* modify ..

* optimize code

* assign TableDto' count default value 0 (fix #995)

* check if table exists when show partitions (fix #1028)

* check table exists when drop partition (fix #1029)

* check if partition name is legal (fix #1022)

* modify status code when partition tag is illegal

* update changlog

* add info to /system url

* add binary index and add bin uri & handler method(not completed)

* optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067)

* fix test partition bug

* fix test bug when check insert records

* add binary vectors test

* add default for offset and page_size

* fix uinttest bug

* [skip ci] remove comments

* optimize web code for PR comments

* add new folder named utils

* check offset and pagesize (fix #1082)

* improve error message if offset or page_size is not legal (fix #1075)

* add log into web module

* update changlog

* check gpu sources setting when assign repeated value (fix #990)

* update changlog

* clang-format pass

* add default handler in http handler

* [skip ci] improve error msg when check gpu resources

* change check offset way

* remove func IsIntStr

* add case

* change int32 to int64 when check number str

* add log in we module(doing)

* update test case

* add log in web controller

* remove surplus dot

* add preload into /system/

* change get_milvus() to get_milvus(args['handler'])

* support load table into memory with http server (fix #1115)

* [skip ci] comment surplus dto in VectorDto

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>

* Fix #1140 (#1162)

* fix

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* update...

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* fix2

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* fix3

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* update changelog

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* Update INSTALL.md (#1175)

* Update INSTALL.md

1. Change image tag and Milvus source code to latest.
2. Fix a typo

Signed-off-by: Lu Wang <yamasite@qq.com>

* Update INSTALL.md

Signed-off-by: lu.wang <yamasite@qq.com>

* add Tanimoto ground truth (#1138)

* add milvus ground truth

* add milvus groundtruth

* [skip ci] add milvus ground truth

* [skip ci]add tanimoto ground truth

* fix mix case bug (#1208)

* fix mix case bug

Signed-off-by: del.zhenwu <zhenxiang.li@zilliz.com>

* Remove case.md

Signed-off-by: del.zhenwu <zhenxiang.li@zilliz.com>

* Update README.md (#1206)

Add LFAI mailing lists.

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Add design.md to store links to design docs (#1219)

* Update README.md

Add link to Milvus design docs

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Create design.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update design.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Add troubleshooting info about libmysqlpp.so.3 error (#1225)

* Update INSTALL.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update INSTALL.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update README.md (#1233)

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* #1240 Update license declaration of each file (#1241)

* #1240 Update license declaration of each files

Signed-off-by: jinhai <hai.jin@zilliz.com>

* #1240 Update CHANGELOG

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Update README.md (#1258)

Add Jenkins master badge.

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update INSTALL.md (#1265)

Fix indentation.

* support CPU profiling (#1251)

* #1250 support CPU profiling

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #1250 fix code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* Fix HNSW crash (#1262)

* fix

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* update.

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* Add troubleshooting information for INSTALL.md and enhance readability (#1274)

* Update INSTALL.md

1. Add new troubleshooting message;
2. Enhance readability.

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update INSTALL.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update INSTALL.md

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Update INSTALL.md

Add CentOS link.

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* Create COMMUNITY.md (#1292)

Signed-off-by: Lutkin Wang <yamasite@qq.com>

* fix gtest

* add copyright

* fix gtest

* MERGE_NOT_YET

* fix lint

Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>
Co-authored-by: BossZou <40255591+BossZou@users.noreply.github.com>
Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Tinkerrr <linxiaojun.cn@outlook.com>
Co-authored-by: del-zhenwu <56623710+del-zhenwu@users.noreply.github.com>
Co-authored-by: Lutkin Wang <yamasite@qq.com>
Co-authored-by: shengjh <46514371+shengjh@users.noreply.github.com>
Co-authored-by: ABNER-1 <ABNER-1@users.noreply.github.com>
Co-authored-by: Jin Hai <hai.jin@zilliz.com>
Co-authored-by: shiyu22 <cshiyu22@gmail.com>

* #1302 Get all record IDs in a segment by given a segment id

* Remove query time ranges

Signed-off-by: zhenwu <zw@zilliz.com>

* #1295 let wal enable by default

* fix cases

Signed-off-by: zhenwu <zw@zilliz.com>

* fix partition cases

Signed-off-by: zhenwu <zw@zilliz.com>

* [skip ci] update test_db

* update

* fix case bug

Signed-off-by: zhenwu <zw@zilliz.com>

* lint

* fix test case failures

* remove some code

* Caiyd crud 1 (#1377)

* fix clang format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest build error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build issue when enable profiling

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix hastable bug

* update bloom filter

* update

* benchmark

* update benchmark

* update

* update

* remove wal record size

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* remove wal record size config

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* update apply deletes: switch to binary search

* update sdk_simple

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update apply deletes: switch to binary search

* add test_search_by_id

Signed-off-by: zhenwu <zw@zilliz.com>

* add more log

* flush error with multi same ids

Signed-off-by: zhenwu <zw@zilliz.com>

* modify wal config

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* update

* add binary search_by_id

* fix case bug

Signed-off-by: zhenwu <zw@zilliz.com>

* update cases

Signed-off-by: zhenwu <zw@zilliz.com>

* fix unit test #1395

* improve merge performance

* add uids_ for VectorIndex to improve search performance

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix error

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

* fix search

* fix record num

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* refine code

* refine code

* Add get_vector_ids test cases (#1407)

* fix order

* add wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix wal case

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix invalid operation issue

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix invalid operation issue

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix bug

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* fix bug

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* crud fix

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* crud fix

Signed-off-by: sahuang <xiaohaix@student.unimelb.edu.au>

* add table info test cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* add to compact case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* add to compact case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* add to compact case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* add case and debug compact

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* test pdb

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* test pdb

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* test pdb

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix cases

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update table_info case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update table_info case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update table_info case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update get vector ids case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update get vector ids case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update get vector ids case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update get vector ids case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* update case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* pdb test

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* pdb test

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* add tests for get_vector_ids

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix case

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* add binary and ip

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix binary index

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* fix pdb

Signed-off-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>

* #1408 fix search result in-correct after DeleteById

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add one case

* delete failed segment

* update serialize

* update serialize

* fix case

Signed-off-by: zhenwu <zw@zilliz.com>

* update

* update case assertion

Signed-off-by: zhenwu <zw@zilliz.com>

* [skip ci] update config

* change bloom filter msync flag to async

* #1319 add more timing debug info

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

* update

* add normalize

Signed-off-by: zhenwu <zw@zilliz.com>

* add normalize

Signed-off-by: zhenwu <zw@zilliz.com>

* add normalize

Signed-off-by: zhenwu <zw@zilliz.com>

* Fix compiling error

Signed-off-by: jinhai <hai.jin@zilliz.com>

* support ip (#1383)

* support ip

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* IP result distance sort by descend

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* update

Signed-off-by: Nicky <nicky.xj.lin@gmail.com>

* format

Signed-off-by: xiaojun.lin <xiaojun.lin@zilliz.com>

* get table lsn

* Remove unused third party

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Refine code

Signed-off-by: jinhai <hai.jin@zilliz.com>

* #1319 fix clang format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix wal applied lsn

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* validate partition tag

* #1319 improve search performance

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* build error

Co-authored-by: Zhiru Zhu <youny626@hotmail.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
Co-authored-by: Xiaohai Xu <xiaohaix@student.unimelb.edu.au>
Co-authored-by: shengjh <46514371+shengjh@users.noreply.github.com>
Co-authored-by: del-zhenwu <56623710+del-zhenwu@users.noreply.github.com>
Co-authored-by: shengjun.li <49774184+shengjun1985@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>
Co-authored-by: BossZou <40255591+BossZou@users.noreply.github.com>
Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Tinkerrr <linxiaojun.cn@outlook.com>
Co-authored-by: Lutkin Wang <yamasite@qq.com>
Co-authored-by: ABNER-1 <ABNER-1@users.noreply.github.com>
Co-authored-by: shiyu22 <cshiyu22@gmail.com>
2020-02-29 16:11:31 +08:00
BossZou c63a50c635
Allow system conf modifiable and some take effect directly #1263 (#1287)
* finish configure modifiable and some take effect runtime

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* move test file to unittet folder

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add test case for cofig

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix format issue

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize callback

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix error when get config by http in CPU mode

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update callback map

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* correct typo

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update callback register

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add identity in registers

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add identity in FaissPass class

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add comments at cmd test if fail

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove some comments and add test for validate storage path

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* format pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove endpoint info and update CORS

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add test case in web module to test drop a non-existent table

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize config store

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add check code in config

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add scheduler gpu handler

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update optimizer faiss pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add default value of gpu_enable_ in GpuCacheMgr

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize code: remove comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* format pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add gpu macro in optimizer handler module to fix compile bug on cpu version

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* change cache setting unit  bytes

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* rename variables

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-02-26 18:54:32 +08:00
Jin Hai e1027e9e65
#1240 Update license declaration of each file (#1241)
* #1240 Update license declaration of each files

Signed-off-by: jinhai <hai.jin@zilliz.com>

* #1240 Update CHANGELOG

Signed-off-by: jinhai <hai.jin@zilliz.com>
2020-02-17 23:40:58 +08:00
BossZou 015f0352a6 Fix http bug & add binary vectors support (#1073)
* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* change gpu resources invoke way to cmd()

* set advanced config name add DEFAULT

* change config setting to cmd

* modify ..

* optimize code

* assign TableDto' count default value 0 (fix #995)

* check if table exists when show partitions (fix #1028)

* check table exists when drop partition (fix #1029)

* check if partition name is legal (fix #1022)

* modify status code when partition tag is illegal

* update changlog

* add info to /system url

* add binary index and add bin uri & handler method(not completed)

* optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067)

* fix test partition bug

* fix test bug when check insert records

* add binary vectors test

* add default for offset and page_size

* fix uinttest bug

* [skip ci] remove comments

* optimize web code for PR comments

* add new folder named utils

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2020-01-18 10:05:49 +08:00
BossZou a0faf1a786 Add http server (#956)
* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* set advanced config name add DEFAULT

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2020-01-13 13:55:48 +08:00
Cai Yudong b9d9b8ecc2 support minio storage (#895)
* #815 add S3 client

* #815 update S3 interface

* #815 add bucket_name_ as class field, add StartService() and StopService()

* #815 add FileIOReader and FileIOWriter

* #815 fix unittest test_storage build error

* #815 add GetObjectStr()

* #815 add S3IOReader/S3IOWriter

* #815 add ListObjects(), support put string with \0, launch S3 service when server start

* #815 read/write s3 index file pass

* #815 add MinIO related config

* #815 fix storage config issuegs

* #815 code clean

* #815 use local file or S3 via minio_enable

* #815 add DeleteObjects() for S3

* #815 print out IO rate when read_index & write_index

* #815 clean code

* #815 fix centos build error

* #815 support MinIO storage

* #815 fix typo
2020-01-04 10:39:52 +08:00
XuPeng-SH d316f18cbc Integrate libfiu for fault injection (#784)
* (feat): add libfiu for fault injection

* (feat): change libfiu from static to shared

* (feat): install libfiu
2019-12-18 18:11:18 +08:00
BossZou 5e045af51f Refactor server module separating Grpc from server handler and scheduler (#757)
* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t
2019-12-12 14:13:22 +08:00
Zhiru Zhu e99896ac62 Add Opentracing (#729)
* add TracerUtil

* Interceptor ok

* add handler

* add context

* minor update

* keep span in trace context

* add span in search okay

* Update Context.cpp

* refactor

* refactor

* refactor

* format

* add context in SearchJob

* trace search okay

* add back finish span in interceptor

* add namespace

* add tracing config in server config

* add random id

* debug mode okay

* update CMakeLists

* add opentracing to cmake

* update unittest

* add tracing namespace

* remove std::run_time error

* add lock when erasing context_map

* update tracing config

* lint

* update CHANGELOG

* small fix

* fix server unit test

* fix scheduler unit test

* fix db unit test

* lint

* fix db unit test gpu version

* rename to tracing_config

* fix

* update

* trigger ci
2019-12-10 15:24:54 +08:00
groot f3ae1bf155 #266 Rpc request source code refactor 2019-11-18 15:10:15 +08:00
Zhiru Zhu 8ae3a83948
Update CMakeLists.txt 2019-11-14 21:12:51 +08:00
Zhiru Zhu 13cb3b8914
Update CMakeLists.txt 2019-11-14 21:02:23 +08:00
Zhiru Zhu bf0e5cbc9d Add option to disable/enble prometheus when compiling 2019-11-14 11:22:07 +08:00
wxyu f251d26bbb Reside src/external in thirdparty 2019-11-13 16:35:37 +08:00
Zhiru Zhu 3c0c4646b0 issue 229, 230 and 237 2019-11-07 15:35:53 +08:00
youny626 bede77751e clang format 2019-11-05 13:35:24 +08:00
youny626 4ca6b59b91 add CPU version 2019-11-05 10:23:59 +08:00