milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Mishards 070 (#1699) 

* update mishards to 0.7.0

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update mishards to 0.7.0

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix search bug and example pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add new api

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add segment stat

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix table info issue

Signed-off-by: yhz <413554850@qq.com>

* fix mishards api issue

Signed-off-by: yhz <413554850@qq.com>

* update mishards

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update all_in_one docker images

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix delete_by_id param parser & remove some comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update yaml config file

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update all_in_one config

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove delete_by_range comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update cmd api

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add warning when search

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update service_handler

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update shrads requiremtns

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* [skip ci] remove surplus log info

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
pull/1720/head
BossZou 2020-03-20 20:21:11 +08:00 committed by GitHub
parent fe2595fa38
commit 3c3617fd81
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
14 changed files with 982 additions and 239 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -28,6 +28,7 @@ Please mark all change in change log and use the issue from GitHub
- \#1649 Fix Milvus crash on old CPU - \#1649 Fix Milvus crash on old CPU
- \#1653 IndexFlat (SSE) and IndexBinaryFlat performance improvement for small NQ - \#1653 IndexFlat (SSE) and IndexBinaryFlat performance improvement for small NQ
- \#1678 Remove CUSTOMIZATION macro - \#1678 Remove CUSTOMIZATION macro
- \#1698 Upgrade mishards to v0.7.0
## Task ## Task

4
shards/all_in_one/all_in_one.yml
View File

@ -3,7 +3,7 @@ services:
milvus_wr: milvus_wr:
runtime: nvidia runtime: nvidia
restart: always restart: always
image: milvusdb/milvus:0.6.0-gpu-d120719-2b40dd image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
ports: ports:
- "0.0.0.0:19540:19530" - "0.0.0.0:19540:19530"
volumes: volumes:
@ -13,7 +13,7 @@ services:
milvus_ro: milvus_ro:
runtime: nvidia runtime: nvidia
restart: always restart: always
image: milvusdb/milvus:0.6.0-gpu-d120719-2b40dd image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
ports: ports:
- "0.0.0.0:19541:19530" - "0.0.0.0:19541:19530"
volumes: volumes:

200
shards/all_in_one/ro_server.yml
View File

@ -1,41 +1,185 @@
# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.2
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config: server_config:
address: 0.0.0.0 # milvus server ip address (IPv4) address: 0.0.0.0
port: 19530 # port range: 1025 ~ 65534 port: 19530
deploy_mode: cluster_readonly # deployment type: single, cluster_readonly, cluster_writable deploy_mode: cluster_readonly
time_zone: UTC+8 time_zone: UTC+8
web_port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_table | A comma-separated list of table names that need to be pre- | StringList | |
# | loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config: db_config:
primary_path: /var/lib/milvus # path used to store data and meta backend_url: sqlite://:@:/
secondary_path: # path used to store data only, split by semicolon preload_table:
auto_flush_interval: 1
backend_url: sqlite://:@:/ # URI format: dialect://username:password@host:port/database #----------------------+------------------------------------------------------------+------------+-----------------+
# Keep 'dialect://:@:/', and replace other texts with real values # Storage Config | Description | Type | Default |
# Replace 'dialect' with 'mysql' or 'sqlite' #----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
insert_buffer_size: 1 # GB, maximum insert buffer size allowed # | index data. | | |
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory #----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload # | to save vector data and index data. | | |
# you can specify preload tables like this: table1,table2,table3 #----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config: metric_config:
enable_monitor: false # enable monitoring or not enable_monitor: false
collector: prometheus # prometheus address: 127.0.0.1
prometheus_config: port: 9091
port: 8080 # port prometheus uses to fetch metrics
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | Integer | 1 (GB) |
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config: cache_config:
cpu_cache_capacity: 4 # GB, CPU memory used for cache cpu_cache_capacity: 4
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered insert_buffer_size: 1
gpu_cache_capacity: 1 # GB, GPU memory used for cache cache_insert_data: false
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
cache_insert_data: false # whether to load inserted data into cache
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config: engine_config:
use_blas_threshold: 800 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times use_blas_threshold: 1100
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times gpu_search_threshold: 1000
resource_config: #----------------------+------------------------------------------------------------+------------+-----------------+
search_resources: # define the GPUs used for search computation, valid value: gpux # GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
enable: true
cache_capacity: 1
search_resources:
- gpu0 - gpu0
index_build_device: gpu0 # GPU used for building index build_index_resources:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal

200
shards/all_in_one/wr_server.yml
View File

@ -1,41 +1,185 @@
# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.2
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config: server_config:
address: 0.0.0.0 # milvus server ip address (IPv4) address: 0.0.0.0
port: 19530 # port range: 1025 ~ 65534 port: 19530
deploy_mode: cluster_writable # deployment type: single, cluster_readonly, cluster_writable deploy_mode: cluster_writable
time_zone: UTC+8 time_zone: UTC+8
web_port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_table | A comma-separated list of table names that need to be pre- | StringList | |
# | loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config: db_config:
primary_path: /var/lib/milvus # path used to store data and meta backend_url: sqlite://:@:/
secondary_path: # path used to store data only, split by semicolon preload_table:
auto_flush_interval: 1
backend_url: sqlite://:@:/ # URI format: dialect://username:password@host:port/database #----------------------+------------------------------------------------------------+------------+-----------------+
# Keep 'dialect://:@:/', and replace other texts with real values # Storage Config | Description | Type | Default |
# Replace 'dialect' with 'mysql' or 'sqlite' #----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
insert_buffer_size: 2 # GB, maximum insert buffer size allowed # | index data. | | |
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory #----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload # | to save vector data and index data. | | |
# you can specify preload tables like this: table1,table2,table3 #----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config: metric_config:
enable_monitor: false # enable monitoring or not enable_monitor: false
collector: prometheus # prometheus address: 127.0.0.1
prometheus_config: port: 9091
port: 8080 # port prometheus uses to fetch metrics
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | Integer | 1 (GB) |
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config: cache_config:
cpu_cache_capacity: 2 # GB, CPU memory used for cache cpu_cache_capacity: 4
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered insert_buffer_size: 1
gpu_cache_capacity: 2 # GB, GPU memory used for cache cache_insert_data: false
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
cache_insert_data: false # whether to load inserted data into cache
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config: engine_config:
use_blas_threshold: 800 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times use_blas_threshold: 1100
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times gpu_search_threshold: 1000
resource_config: #----------------------+------------------------------------------------------------+------------+-----------------+
search_resources: # define the GPUs used for search computation, valid value: gpux # GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
enable: true
cache_capacity: 1
search_resources:
- gpu0 - gpu0
index_build_device: gpu0 # GPU used for building index build_index_resources:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal

4
shards/all_in_one_with_mysql/all_in_one.yml
View File

@ -18,7 +18,7 @@ services:
milvus_wr: milvus_wr:
runtime: nvidia runtime: nvidia
restart: always restart: always
image: milvusdb/milvus:0.6.0-gpu-d120719-2b40dd image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
volumes: volumes:
- /tmp/milvus/db:/var/lib/milvus/db - /tmp/milvus/db:/var/lib/milvus/db
- ./wr_server.yml:/var/lib/milvus/conf/server_config.yaml - ./wr_server.yml:/var/lib/milvus/conf/server_config.yaml
@ -29,7 +29,7 @@ services:
milvus_ro: milvus_ro:
runtime: nvidia runtime: nvidia
restart: always restart: always
image: milvusdb/milvus:0.6.0-gpu-d120719-2b40dd image: milvusdb/milvus:0.7.0-gpu-d031120-40c7e8
volumes: volumes:
- /tmp/milvus/db:/var/lib/milvus/db - /tmp/milvus/db:/var/lib/milvus/db
- ./ro_server.yml:/var/lib/milvus/conf/server_config.yaml - ./ro_server.yml:/var/lib/milvus/conf/server_config.yaml

201
shards/all_in_one_with_mysql/ro_server.yml
View File

@ -1,43 +1,186 @@
version: 0.1 # Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.2
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config: server_config:
address: 0.0.0.0 # milvus server ip address (IPv4) address: 0.0.0.0
port: 19530 # port range: 1025 ~ 65534 port: 19530
deploy_mode: cluster_readonly # deployment type: single, cluster_readonly, cluster_writable deploy_mode: cluster_readonly
time_zone: UTC+8 time_zone: UTC+8
web_port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_table | A comma-separated list of table names that need to be pre- | StringList | |
# | loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config: db_config:
primary_path: /var/lib/milvus # path used to store data and meta
secondary_path: # path used to store data only, split by semicolon
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
# URI format: dialect://username:password@host:port/database preload_table:
# Keep 'dialect://:@:/', and replace other texts with real values auto_flush_interval: 1
# Replace 'dialect' with 'mysql' or 'sqlite'
insert_buffer_size: 1 # GB, maximum insert buffer size allowed #----------------------+------------------------------------------------------------+------------+-----------------+
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory # Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload # primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
# you can specify preload tables like this: table1,table2,table3 # | index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
# | to save vector data and index data. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config: metric_config:
enable_monitor: false # enable monitoring or not enable_monitor: false
collector: prometheus # prometheus address: 127.0.0.1
prometheus_config: port: 9091
port: 8080 # port prometheus uses to fetch metrics
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | Integer | 1 (GB) |
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config: cache_config:
cpu_cache_capacity: 4 # GB, CPU memory used for cache cpu_cache_capacity: 4
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered insert_buffer_size: 1
gpu_cache_capacity: 1 # GB, GPU memory used for cache cache_insert_data: false
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
cache_insert_data: false # whether to load inserted data into cache
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config: engine_config:
use_blas_threshold: 800 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times use_blas_threshold: 1100
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times gpu_search_threshold: 1000
resource_config: #----------------------+------------------------------------------------------------+------------+-----------------+
search_resources: # define the GPUs used for search computation, valid value: gpux # GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
enable: true
cache_capacity: 1
search_resources:
- gpu0 - gpu0
index_build_device: gpu0 # GPU used for building index build_index_resources:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal

202
shards/all_in_one_with_mysql/wr_server.yml
View File

@ -1,42 +1,186 @@
version: 0.1 # Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.2
#----------------------+------------------------------------------------------------+------------+-----------------+
# Server Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# deploy_mode | Milvus deployment type: | DeployMode | single |
# | single, cluster_readonly, cluster_writable | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# time_zone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# web_port | Port that Milvus web server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
server_config: server_config:
address: 0.0.0.0 # milvus server ip address (IPv4) address: 0.0.0.0
port: 19530 # port range: 1025 ~ 65534 port: 19530
deploy_mode: cluster_writable # deployment type: single, cluster_readonly, cluster_writable deploy_mode: cluster_writable
time_zone: UTC+8 time_zone: UTC+8
web_port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# DataBase Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# backend_url | URL for metadata storage, using SQLite (for single server | URL | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_table | A comma-separated list of table names that need to be pre- | StringList | |
# | loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
db_config: db_config:
primary_path: /var/lib/milvus # path used to store data and meta backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus
secondary_path: # path used to store data only, split by semicolon preload_table:
auto_flush_interval: 1
backend_url: mysql://root:milvusroot@milvus-mysql:3306/milvus # URI format: dialect://username:password@host:port/database #----------------------+------------------------------------------------------------+------------+-----------------+
# Keep 'dialect://:@:/', and replace other texts with real values # Storage Config | Description | Type | Default |
# Replace 'dialect' with 'mysql' or 'sqlite' #----------------------+------------------------------------------------------------+------------+-----------------+
# primary_path | Primary directory used to save meta data, vector data and | Path | /var/lib/milvus |
insert_buffer_size: 2 # GB, maximum insert buffer size allowed # | index data. | | |
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory #----------------------+------------------------------------------------------------+------------+-----------------+
# secondary_path | A semicolon-separated list of secondary directories used | Path | |
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload # | to save vector data and index data. | | |
# you can specify preload tables like this: table1,table2,table3 #----------------------+------------------------------------------------------------+------------+-----------------+
storage_config:
primary_path: /var/lib/milvus
secondary_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable_monitor | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric_config: metric_config:
enable_monitor: false # enable monitoring or not enable_monitor: false
collector: prometheus # prometheus address: 127.0.0.1
prometheus_config: port: 9091
port: 8080 # port prometheus uses to fetch metrics
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cpu_cache_capacity | The size of CPU memory used for caching data for faster | Integer | 4 (GB) |
# | query. The sum of 'cpu_cache_capacity' and | | |
# | 'insert_buffer_size' must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | Integer | 1 (GB) |
# | The sum of 'insert_buffer_size' and 'cpu_cache_capacity' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_insert_data | Whether to load data to cache for hot query | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache_config: cache_config:
cpu_cache_capacity: 2 # GB, CPU memory used for cache cpu_cache_capacity: 4
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered insert_buffer_size: 1
gpu_cache_capacity: 2 # GB, GPU memory used for cache cache_insert_data: false
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
cache_insert_data: false # whether to load inserted data into cache
#----------------------+------------------------------------------------------------+------------+-----------------+
# Engine Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# use_blas_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1100 |
# | compared with 'nq' to decide if OpenBLAS should be used. | | |
# | If nq >= use_blas_threshold, OpenBLAS will be used, search | | |
# | response time will be stable but the search speed will be | | |
# | slower; if nq < use_blas_threshold, AVX or SSE will be | | |
# | used, search speed will be faster but search response time | | |
# | will fluctuate. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
engine_config: engine_config:
use_blas_threshold: 800 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times use_blas_threshold: 1100
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times gpu_search_threshold: 1000
resource_config: #----------------------+------------------------------------------------------------+------------+-----------------+
search_resources: # define the GPUs used for search computation, valid value: gpux # GPU Resource Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable GPU resources or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_capacity | The size of GPU memory per card used for cache. | Integer | 1 (GB) |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_resources | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_resources| The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu_resource_config:
enable: true
cache_capacity: 1
search_resources:
- gpu0 - gpu0
index_build_device: gpu0 # GPU used for building index build_index_resources:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Tracing Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# json_config_path | Absolute path for tracing config file. | Path | |
# | Leave it empty, a no-op tracer will be created. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
tracing_config:
json_config_path:
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# recovery_error_ignore| Whether to ignore logs with errors that happens during WAL | Boolean | true |
# | recovery. If true, when Milvus restarts for recovery and | | |
# | there are errors in WAL log files, log files with errors | | |
# | are ignored. If false, Milvus does not restart when there | | |
# | are errors in WAL log files. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# buffer_size | Sum total of the read buffer and the write buffer in MBs. | Integer | 256 (MB) |
# | buffer_size must be in range [64, 4096] (MB). | | |
# | If the value you specified is out of range, Milvus | | |
# | automatically uses the boundary value closest to the | | |
# | specified value. It is recommended you set buffer_size to | | |
# | a value greater than the inserted data size of a single | | |
# | insert operation for better performance. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# wal_path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal_config:
enable: true
recovery_error_ignore: true
buffer_size: 256
wal_path: /var/lib/milvus/wal

1
shards/mishards/exception_codes.py
View File

@ -8,3 +8,4 @@ TABLE_NOT_FOUND_CODE = 20001
INVALID_ARGUMENT_CODE = 20002 INVALID_ARGUMENT_CODE = 20002
INVALID_DATE_RANGE_CODE = 20003 INVALID_DATE_RANGE_CODE = 20003
INVALID_TOPK_CODE = 20004 INVALID_TOPK_CODE = 20004
SEARCH_PARAM_LOSS_CODE = 20005

4
shards/mishards/exceptions.py
View File

@ -30,6 +30,10 @@ class InvalidTopKError(BaseException):
code = codes.INVALID_TOPK_CODE code = codes.INVALID_TOPK_CODE
class SearchParamError(BaseException):
code = codes.SEARCH_PARAM_LOSS_CODE
class InvalidArgumentError(BaseException): class InvalidArgumentError(BaseException):
code = codes.INVALID_ARGUMENT_CODE code = codes.INVALID_ARGUMENT_CODE

77
shards/mishards/grpc_utils/grpc_args_parser.py
View File

@ -1,3 +1,4 @@
import ujson
from milvus import Status from milvus import Status
from functools import wraps from functools import wraps
@ -21,26 +22,30 @@ class GrpcArgsParser(object):
@error_status @error_status
def parse_proto_TableSchema(cls, param): def parse_proto_TableSchema(cls, param):
_table_schema = { _table_schema = {
'status': param.status, 'collection_name': param.table_name,
'table_name': param.table_name,
'dimension': param.dimension, 'dimension': param.dimension,
'index_file_size': param.index_file_size, 'index_file_size': param.index_file_size,
'metric_type': param.metric_type 'metric_type': param.metric_type
} }
return _table_schema return param.status, _table_schema
@classmethod @classmethod
@error_status @error_status
def parse_proto_TableName(cls, param): def parse_proto_TableName(cls, param):
return param.table_name return param.table_name
@classmethod
@error_status
def parse_proto_FlushParam(cls, param):
return list(param.table_name_array)
@classmethod @classmethod
@error_status @error_status
def parse_proto_Index(cls, param): def parse_proto_Index(cls, param):
_index = { _index = {
'index_type': param.index_type, 'index_type': param.index_type,
'nlist': param.nlist 'params': param.extra_params[0].value
} }
return _index return _index
@ -49,12 +54,14 @@ class GrpcArgsParser(object):
@error_status @error_status
def parse_proto_IndexParam(cls, param): def parse_proto_IndexParam(cls, param):
_table_name = param.table_name _table_name = param.table_name
_status, _index = cls.parse_proto_Index(param.index) _index_type = param.index_type
_index_param = {}
if not _status.OK(): for params in param.extra_params:
raise Exception("Argument parse error") if params.key == 'params':
_index_param = ujson.loads(str(params.value))
return _table_name, _index return _table_name, _index_type, _index_param
@classmethod @classmethod
@error_status @error_status
@ -63,14 +70,6 @@ class GrpcArgsParser(object):
return _cmd return _cmd
@classmethod
@error_status
def parse_proto_Range(cls, param):
_start_value = param.start_value
_end_value = param.end_value
return _start_value, _end_value
@classmethod @classmethod
@error_status @error_status
def parse_proto_RowRecord(cls, param): def parse_proto_RowRecord(cls, param):
@ -79,32 +78,52 @@ class GrpcArgsParser(object):
@classmethod @classmethod
def parse_proto_PartitionParam(cls, param): def parse_proto_PartitionParam(cls, param):
_table_name = param.table_name _table_name = param.table_name
_partition_name = param.partition_name
_tag = param.tag _tag = param.tag
return _table_name, _partition_name, _tag return _table_name, _tag
@classmethod @classmethod
@error_status @error_status
def parse_proto_SearchParam(cls, param): def parse_proto_SearchParam(cls, param):
_table_name = param.table_name _table_name = param.table_name
_topk = param.topk _topk = param.topk
_nprobe = param.nprobe
_status, _range = cls.parse_proto_Range(param.query_range_array)
if not _status.OK(): if len(param.extra_params) == 0:
raise Exception("Argument parse error") raise Exception("Search param loss")
_params = ujson.loads(str(param.extra_params[0].value))
_row_record = param.query_record_array _query_record_array = []
if param.query_record_array:
for record in param.query_record_array:
if record.float_data:
_query_record_array.append(list(record.float_data))
else:
_query_record_array.append(bytes(record.binary_data))
else:
raise Exception("Search argument parse error: record array is empty")
return _table_name, _row_record, _range, _topk return _table_name, _query_record_array, _topk, _params
@classmethod @classmethod
@error_status @error_status
def parse_proto_DeleteByRangeParam(cls, param): def parse_proto_DeleteByIDParam(cls, param):
_table_name = param.table_name _table_name = param.table_name
_range = param.range _id_array = list(param.id_array)
_start_value = _range.start_value
_end_value = _range.end_value
return _table_name, _start_value, _end_value return _table_name, _id_array
@classmethod
@error_status
def parse_proto_VectorIdentity(cls, param):
_table_name = param.table_name
_id = param.id
return _table_name, _id
@classmethod
@error_status
def parse_proto_GetVectorIDsParam(cls, param):
_table__name = param.table_name
_segment_name = param.segment_name
return _table__name, _segment_name

5
shards/mishards/models.py
View File

@ -23,6 +23,7 @@ class TableFiles(db.Model):
id = Column(BigInteger, primary_key=True, autoincrement=True) id = Column(BigInteger, primary_key=True, autoincrement=True)
table_id = Column(String(50)) table_id = Column(String(50))
segment_id = Column(String(50))
engine_type = Column(Integer) engine_type = Column(Integer)
file_id = Column(String(50)) file_id = Column(String(50))
file_type = Column(Integer) file_type = Column(Integer)
@ -31,6 +32,7 @@ class TableFiles(db.Model):
updated_time = Column(BigInteger) updated_time = Column(BigInteger)
created_on = Column(BigInteger) created_on = Column(BigInteger)
date = Column(Integer) date = Column(Integer)
flush_lsn = Column(Integer)
table = relationship( table = relationship(
'Tables', 'Tables',
@ -55,9 +57,10 @@ class Tables(db.Model):
created_on = Column(Integer) created_on = Column(Integer)
flag = Column(Integer, default=0) flag = Column(Integer, default=0)
index_file_size = Column(Integer) index_file_size = Column(Integer)
index_params = Column(String(50))
engine_type = Column(Integer) engine_type = Column(Integer)
nlist = Column(Integer)
metric_type = Column(Integer) metric_type = Column(Integer)
flush_lsn = Column(Integer)
def files_to_search(self, date_range=None): def files_to_search(self, date_range=None):
cond = or_( cond = or_(

315
shards/mishards/service_handler.py
View File

@ -3,13 +3,14 @@ import time
import datetime import datetime
import json import json
from collections import defaultdict from collections import defaultdict
import ujson
import multiprocessing import multiprocessing
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from milvus.grpc_gen import milvus_pb2, milvus_pb2_grpc, status_pb2 from milvus.grpc_gen import milvus_pb2, milvus_pb2_grpc, status_pb2
from milvus.grpc_gen.milvus_pb2 import TopKQueryResult from milvus.grpc_gen.milvus_pb2 import TopKQueryResult
from milvus.client.abstract import Range
from milvus.client import types as Types from milvus.client import types as Types
from milvus import MetricType
from mishards import (db, settings, exceptions) from mishards import (db, settings, exceptions)
from mishards.grpc_utils import mark_grpc_method from mishards.grpc_utils import mark_grpc_method
@ -106,21 +107,20 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
table_meta, table_meta,
vectors, vectors,
topk, topk,
nprobe, search_params,
range_array=None,
partition_tags=None, partition_tags=None,
**kwargs): **kwargs):
metadata = kwargs.get('metadata', None) metadata = kwargs.get('metadata', None)
range_array = [ # range_array = [
utilities.range_to_date(r, metadata=metadata) for r in range_array # utilities.range_to_date(r, metadata=metadata) for r in range_array
] if range_array else None # ] if range_array else None
routing = {} routing = {}
p_span = None if self.tracer.empty else context.get_active_span( p_span = None if self.tracer.empty else context.get_active_span(
).context ).context
with self.tracer.start_span('get_routing', child_of=p_span): with self.tracer.start_span('get_routing', child_of=p_span):
routing = self.router.routing(table_id, routing = self.router.routing(table_id,
range_array=range_array, # range_array=range_array,
partition_tags=partition_tags, partition_tags=partition_tags,
metadata=metadata) metadata=metadata)
logger.info('Routing: {}'.format(routing)) logger.info('Routing: {}'.format(routing))
@ -130,10 +130,10 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
rs = [] rs = []
all_topk_results = [] all_topk_results = []
def search(addr, table_id, file_ids, vectors, topk, nprobe, **kwargs): def search(addr, table_id, file_ids, vectors, topk, params, **kwargs):
logger.info( logger.info(
'Send Search Request: addr={};table_id={};ids={};nq={};topk={};nprobe={}' 'Send Search Request: addr={};table_id={};ids={};nq={};topk={};params={}'
.format(addr, table_id, file_ids, len(vectors), topk, nprobe)) .format(addr, table_id, file_ids, len(vectors), topk, params))
conn = self.router.query_conn(addr, metadata=metadata) conn = self.router.query_conn(addr, metadata=metadata)
start = time.time() start = time.time()
@ -143,11 +143,14 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
with self.tracer.start_span('search_{}'.format(addr), with self.tracer.start_span('search_{}'.format(addr),
child_of=span): child_of=span):
ret = conn.conn.search_vectors_in_files(table_name=table_id, ret = conn.conn.search_vectors_in_files(collection_name=table_id,
file_ids=file_ids, file_ids=file_ids,
query_records=vectors, query_records=vectors,
top_k=topk, top_k=topk,
nprobe=nprobe) params=params)
if ret.status.error_code != 0:
logger.error(ret.status)
end = time.time() end = time.time()
all_topk_results.append(ret) all_topk_results.append(ret)
@ -155,14 +158,14 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
with self.tracer.start_span('do_search', child_of=p_span) as span: with self.tracer.start_span('do_search', child_of=p_span) as span:
with ThreadPoolExecutor(max_workers=self.max_workers) as pool: with ThreadPoolExecutor(max_workers=self.max_workers) as pool:
for addr, params in routing.items(): for addr, params in routing.items():
for table_id, file_ids in params.items(): for sub_table_id, file_ids in params.items():
res = pool.submit(search, res = pool.submit(search,
addr, addr,
table_id, table_id,
file_ids, file_ids,
vectors, vectors,
topk, topk,
nprobe, search_params,
span=span) span=span)
rs.append(res) rs.append(res)
@ -177,17 +180,22 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
metadata=metadata) metadata=metadata)
def _create_table(self, table_schema): def _create_table(self, table_schema):
return self.router.connection().create_table(table_schema) return self.router.connection().create_collection(table_schema)
@mark_grpc_method @mark_grpc_method
def CreateTable(self, request, context): def CreateTable(self, request, context):
_status, _table_schema = Parser.parse_proto_TableSchema(request) _status, unpacks = Parser.parse_proto_TableSchema(request)
if not _status.OK(): if not _status.OK():
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
logger.info('CreateTable {}'.format(_table_schema['table_name'])) _status, _table_schema = unpacks
# if _status.error_code == 0:
# logging.warning('[CreateTable] table schema error occurred: {}'.format(_status))
# return _status
logger.info('CreateTable {}'.format(_table_schema['collection_name']))
_status = self._create_table(_table_schema) _status = self._create_table(_table_schema)
@ -195,7 +203,7 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
reason=_status.message) reason=_status.message)
def _has_table(self, table_name, metadata=None): def _has_table(self, table_name, metadata=None):
return self.router.connection(metadata=metadata).has_table(table_name) return self.router.connection(metadata=metadata).has_collection(table_name)
@mark_grpc_method @mark_grpc_method
def HasTable(self, request, context): def HasTable(self, request, context):
@ -217,14 +225,14 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
@mark_grpc_method @mark_grpc_method
def CreatePartition(self, request, context): def CreatePartition(self, request, context):
_table_name, _partition_name, _tag = Parser.parse_proto_PartitionParam(request) _table_name, _tag = Parser.parse_proto_PartitionParam(request)
_status = self.router.connection().create_partition(_table_name, _partition_name, _tag) _status = self.router.connection().create_partition(_table_name, _tag)
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
@mark_grpc_method @mark_grpc_method
def DropPartition(self, request, context): def DropPartition(self, request, context):
_table_name, _partition_name, _tag = Parser.parse_proto_PartitionParam(request) _table_name, _tag = Parser.parse_proto_PartitionParam(request)
_status = self.router.connection().drop_partition(_table_name, _tag) _status = self.router.connection().drop_partition(_table_name, _tag)
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
@ -244,13 +252,10 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
return milvus_pb2.PartitionList(status=status_pb2.Status( return milvus_pb2.PartitionList(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), error_code=_status.code, reason=_status.message),
partition_array=[milvus_pb2.PartitionParam(table_name=param.table_name, partition_tag_array=[param.tag for param in partition_array])
tag=param.tag,
partition_name=param.partition_name)
for param in partition_array])
def _delete_table(self, table_name): def _delete_table(self, table_name):
return self.router.connection().delete_table(table_name) return self.router.connection().drop_collection(table_name)
@mark_grpc_method @mark_grpc_method
def DropTable(self, request, context): def DropTable(self, request, context):
@ -267,8 +272,8 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
def _create_index(self, table_name, index): def _create_index(self, table_name, index_type, param):
return self.router.connection().create_index(table_name, index) return self.router.connection().create_index(table_name, index_type, param)
@mark_grpc_method @mark_grpc_method
def CreateIndex(self, request, context): def CreateIndex(self, request, context):
@ -278,12 +283,12 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
_table_name, _index = unpacks _table_name, _index_type, _index_param = unpacks
logger.info('CreateIndex {}'.format(_table_name)) logger.info('CreateIndex {}'.format(_table_name))
# TODO: interface create_table incompleted # TODO: interface create_table incompleted
_status = self._create_index(_table_name, _index) _status = self._create_index(_table_name, _index_type, _index_param)
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
@ -305,19 +310,22 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
@mark_grpc_method @mark_grpc_method
def Search(self, request, context): def Search(self, request, context):
metadata = {'resp_class': milvus_pb2.TopKQueryResult}
table_name = request.table_name table_name = request.table_name
topk = request.topk topk = request.topk
nprobe = request.nprobe
logger.info('Search {}: topk={} nprobe={}'.format( if len(request.extra_params) == 0:
table_name, topk, nprobe)) raise exceptions.SearchParamError(message="Search parma loss", metadata=metadata)
params = ujson.loads(str(request.extra_params[0].value))
metadata = {'resp_class': milvus_pb2.TopKQueryResult} logger.info('Search {}: topk={} params={}'.format(
table_name, topk, params))
if nprobe > self.MAX_NPROBE or nprobe <= 0: # if nprobe > self.MAX_NPROBE or nprobe <= 0:
raise exceptions.InvalidArgumentError( # raise exceptions.InvalidArgumentError(
message='Invalid nprobe: {}'.format(nprobe), metadata=metadata) # message='Invalid nprobe: {}'.format(nprobe), metadata=metadata)
if topk > self.MAX_TOPK or topk <= 0: if topk > self.MAX_TOPK or topk <= 0:
raise exceptions.InvalidTopKError( raise exceptions.InvalidTopKError(
@ -327,7 +335,7 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
if not table_meta: if not table_meta:
status, info = self.router.connection( status, info = self.router.connection(
metadata=metadata).describe_table(table_name) metadata=metadata).describe_collection(table_name)
if not status.OK(): if not status.OK():
raise exceptions.TableNotFoundError(table_name, raise exceptions.TableNotFoundError(table_name,
metadata=metadata) metadata=metadata)
@ -338,27 +346,24 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
start = time.time() start = time.time()
query_record_array = [] query_record_array = []
if int(table_meta.metric_type) >= MetricType.HAMMING.value:
for query_record in request.query_record_array: for query_record in request.query_record_array:
query_record_array.append(list(query_record.vector_data)) query_record_array.append(bytes(query_record.binary_data))
else:
query_range_array = [] for query_record in request.query_record_array:
for query_range in request.query_range_array: query_record_array.append(list(query_record.float_data))
query_range_array.append(
Range(query_range.start_value, query_range.end_value))
status, id_results, dis_results = self._do_query(context, status, id_results, dis_results = self._do_query(context,
table_name, table_name,
table_meta, table_meta,
query_record_array, query_record_array,
topk, topk,
nprobe, params,
query_range_array,
partition_tags=getattr(request, "partition_tag_array", []), partition_tags=getattr(request, "partition_tag_array", []),
metadata=metadata) metadata=metadata)
now = time.time() now = time.time()
# logger.info('SearchVector takes: {}'.format(now - start)) logger.info('SearchVector takes: {}'.format(now - start))
topk_result_list = milvus_pb2.TopKQueryResult( topk_result_list = milvus_pb2.TopKQueryResult(
status=status_pb2.Status(error_code=status.error_code, status=status_pb2.Status(error_code=status.error_code,
@ -373,7 +378,7 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
raise NotImplemented() raise NotImplemented()
def _describe_table(self, table_name, metadata=None): def _describe_table(self, table_name, metadata=None):
return self.router.connection(metadata=metadata).describe_table(table_name) return self.router.connection(metadata=metadata).describe_collection(table_name)
@mark_grpc_method @mark_grpc_method
def DescribeTable(self, request, context): def DescribeTable(self, request, context):
@ -405,9 +410,53 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
reason=_status.message), reason=_status.message),
) )
def _table_info(self, table_name, metadata=None):
return self.router.connection(metadata=metadata).collection_info(table_name)
@mark_grpc_method
def ShowTableInfo(self, request, context):
_status, _table_name = Parser.parse_proto_TableName(request)
if not _status.OK():
return milvus_pb2.TableInfo(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), )
metadata = {'resp_class': milvus_pb2.TableInfo}
logger.info('ShowTableInfo {}'.format(_table_name))
_status, _info = self._table_info(metadata=metadata, table_name=_table_name)
if _status.OK():
_table_info = milvus_pb2.TableInfo(
status=status_pb2.Status(error_code=_status.code,
reason=_status.message),
total_row_count=_info.count
)
for par_stat in _info.partitions_stat:
_par = milvus_pb2.PartitionStat(
tag=par_stat.tag,
total_row_count=par_stat.count
)
for seg_stat in par_stat.segments_stat:
_par.segments_stat.add(
segment_name=seg_stat.segment_name,
row_count=seg_stat.count,
index_name=seg_stat.index_name,
data_size=seg_stat.data_size,
)
_table_info.partitions_stat.append(_par)
return _table_info
return milvus_pb2.TableInfo(
status=status_pb2.Status(error_code=_status.code,
reason=_status.message),
)
def _count_table(self, table_name, metadata=None): def _count_table(self, table_name, metadata=None):
return self.router.connection( return self.router.connection(
metadata=metadata).get_table_row_count(table_name) metadata=metadata).count_collection(table_name)
@mark_grpc_method @mark_grpc_method
def CountTable(self, request, context): def CountTable(self, request, context):
@ -432,6 +481,9 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
def _get_server_version(self, metadata=None): def _get_server_version(self, metadata=None):
return self.router.connection(metadata=metadata).server_version() return self.router.connection(metadata=metadata).server_version()
def _cmd(self, cmd, metadata=None):
return self.router.connection(metadata=metadata)._cmd(cmd)
@mark_grpc_method @mark_grpc_method
def Cmd(self, request, context): def Cmd(self, request, context):
_status, _cmd = Parser.parse_proto_Command(request) _status, _cmd = Parser.parse_proto_Command(request)
@ -449,18 +501,19 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
error_code=status_pb2.SUCCESS), error_code=status_pb2.SUCCESS),
string_reply=json.dumps(stats, indent=2)) string_reply=json.dumps(stats, indent=2))
if _cmd == 'version': # if _cmd == 'version':
_status, _reply = self._get_server_version(metadata=metadata) # _status, _reply = self._get_server_version(metadata=metadata)
else: # else:
_status, _reply = self.router.connection( # _status, _reply = self.router.connection(
metadata=metadata).server_status() # metadata=metadata).server_status()
_status, _reply = self._cmd(_cmd, metadata=metadata)
return milvus_pb2.StringReply(status=status_pb2.Status( return milvus_pb2.StringReply(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), error_code=_status.code, reason=_status.message),
string_reply=_reply) string_reply=_reply)
def _show_tables(self, metadata=None): def _show_tables(self, metadata=None):
return self.router.connection(metadata=metadata).show_tables() return self.router.connection(metadata=metadata).show_collections()
@mark_grpc_method @mark_grpc_method
def ShowTables(self, request, context): def ShowTables(self, request, context):
@ -472,30 +525,8 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
error_code=_status.code, reason=_status.message), error_code=_status.code, reason=_status.message),
table_names=_results) table_names=_results)
def _delete_by_range(self, table_name, start_date, end_date):
return self.router.connection().delete_vectors_by_range(table_name,
start_date,
end_date)
@mark_grpc_method
def DeleteByRange(self, request, context):
_status, unpacks = \
Parser.parse_proto_DeleteByRangeParam(request)
if not _status.OK():
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
_table_name, _start_date, _end_date = unpacks
logger.info('DeleteByRange {}: {} {}'.format(_table_name, _start_date,
_end_date))
_status = self._delete_by_range(_table_name, _start_date, _end_date)
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
def _preload_table(self, table_name): def _preload_table(self, table_name):
return self.router.connection().preload_table(table_name) return self.router.connection().preload_collection(table_name)
@mark_grpc_method @mark_grpc_method
def PreloadTable(self, request, context): def PreloadTable(self, request, context):
@ -531,13 +562,89 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
return milvus_pb2.IndexParam(status=status_pb2.Status( return milvus_pb2.IndexParam(status=status_pb2.Status(
error_code=_status.code, reason=_status.message)) error_code=_status.code, reason=_status.message))
_index = milvus_pb2.Index(index_type=_index_param._index_type, _index_type = _index_param._index_type
nlist=_index_param._nlist)
return milvus_pb2.IndexParam(status=status_pb2.Status( grpc_index = milvus_pb2.IndexParam(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), error_code=_status.code, reason=_status.message),
table_name=_table_name, table_name=_table_name, index_type=_index_type)
index=_index)
grpc_index.extra_params.add(key='params', value=ujson.dumps(_index_param._params))
return grpc_index
def _get_vector_by_id(self, table_name, vec_id, metadata):
return self.router.connection(metadata=metadata).get_vector_by_id(table_name, vec_id)
@mark_grpc_method
def GetVectorByID(self, request, context):
_status, unpacks = Parser.parse_proto_VectorIdentity(request)
if not _status.OK():
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
metadata = {'resp_class': milvus_pb2.VectorData}
_table_name, _id = unpacks
logger.info('GetVectorByID {}'.format(_table_name))
_status, vector = self._get_vector_by_id(_table_name, _id, metadata)
if not vector:
return milvus_pb2.VectorData(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), )
if isinstance(vector, bytes):
records = milvus_pb2.RowRecord(binary_data=vector)
else:
records = milvus_pb2.RowRecord(float_data=vector)
return milvus_pb2.VectorData(status=status_pb2.Status(
error_code=_status.code, reason=_status.message),
vector_data=records
)
def _get_vector_ids(self, table_name, segment_name, metadata):
return self.router.connection(metadata=metadata).get_vector_ids(table_name, segment_name)
@mark_grpc_method
def GetVectorIDs(self, request, context):
_status, unpacks = Parser.parse_proto_GetVectorIDsParam(request)
if not _status.OK():
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
metadata = {'resp_class': milvus_pb2.VectorIds}
_table_name, _segment_name = unpacks
logger.info('GetVectorIDs {}'.format(_table_name))
_status, ids = self._get_vector_ids(_table_name, _segment_name, metadata)
if not ids:
return milvus_pb2.VectorIds(status=status_pb2.Status(
error_code=_status.code, reason=_status.message), )
return milvus_pb2.VectorIds(status=status_pb2.Status(
error_code=_status.code, reason=_status.message),
vector_id_array=ids
)
def _delete_by_id(self, table_name, id_array):
return self.router.connection().delete_by_id(table_name, id_array)
@mark_grpc_method
def DeleteByID(self, request, context):
_status, unpacks = Parser.parse_proto_DeleteByIDParam(request)
if not _status.OK():
logging.error('DeleteByID {}'.format(_status.message))
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
_table_name, _ids = unpacks
logger.info('DeleteByID {}'.format(_table_name))
_status = self._delete_by_id(_table_name, _ids)
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
def _drop_index(self, table_name): def _drop_index(self, table_name):
return self.router.connection().drop_index(table_name) return self.router.connection().drop_index(table_name)
@ -554,3 +661,35 @@ class ServiceHandler(milvus_pb2_grpc.MilvusServiceServicer):
_status = self._drop_index(_table_name) _status = self._drop_index(_table_name)
return status_pb2.Status(error_code=_status.code, return status_pb2.Status(error_code=_status.code,
reason=_status.message) reason=_status.message)
def _flush(self, table_names):
return self.router.connection().flush(table_names)
@mark_grpc_method
def Flush(self, request, context):
_status, _table_names = Parser.parse_proto_FlushParam(request)
if not _status.OK():
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
logger.info('Flush {}'.format(_table_names))
_status = self._flush(_table_names)
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
def _compact(self, table_name):
return self.router.connection().compact(table_name)
@mark_grpc_method
def Compact(self, request, context):
_status, _table_name = Parser.parse_proto_TableName(request)
if not _status.OK():
return status_pb2.Status(error_code=_status.code,
reason=_status.message)
logger.info('Compact {}'.format(_table_name))
_status = self._compact(_table_name)
return status_pb2.Status(error_code=_status.code,
reason=_status.message)

2
shards/mishards/settings.py
View File

@ -12,7 +12,7 @@ else:
env.read_env() env.read_env()
SERVER_VERSIONS = ['0.6.0'] SERVER_VERSIONS = ['0.7.0']
DEBUG = env.bool('DEBUG', False) DEBUG = env.bool('DEBUG', False)
MAX_RETRY = env.int('MAX_RETRY', 3) MAX_RETRY = env.int('MAX_RETRY', 3)

5
shards/requirements.txt
View File

@ -14,8 +14,8 @@ py==1.8.0
pyasn1==0.4.7 pyasn1==0.4.7
pyasn1-modules==0.2.6 pyasn1-modules==0.2.6
pylint==2.3.1 pylint==2.3.1
#pymilvus==0.2.5 #pymilvus==0.2.8
pymilvus-test==0.2.42 pymilvus-test==0.3.3
pyparsing==2.4.0 pyparsing==2.4.0
pytest==4.6.3 pytest==4.6.3
pytest-level==0.1.1 pytest-level==0.1.1
@ -30,6 +30,7 @@ requests-oauthlib==1.2.0
rsa==4.0 rsa==4.0
six==1.12.0 six==1.12.0
SQLAlchemy==1.3.5 SQLAlchemy==1.3.5
ujson==2.0.1
urllib3==1.25.8 urllib3==1.25.8
jaeger-client>=3.4.0 jaeger-client>=3.4.0
grpcio-opentracing>=1.0 grpcio-opentracing>=1.0