milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

use envoy and headless service to get `low latency、high throughput` support (#4544) 

* init deploy

Signed-off-by: sunyaofei <sunyaofei@ke.com>

* deploy with envoy

Signed-off-by: sunyaofei <sunyaofei@ke.com>

* fix markdown format issues

Signed-off-by: sunyaofei <sunyaofei@ke.com>

* configuration piece error fix

Signed-off-by: sunyaofei <sunyaofei@ke.com>

Co-authored-by: sunyaofei <sunyaofei@ke.com>
pull/4592/head
sunyaofei 2021-01-06 11:52:37 +08:00 committed by GitHub
parent 83800eb232
commit 1d6668156c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
12 changed files with 1249 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
deploy/README.md Normal file
View File

@ -0,0 +1,171 @@
# A solution to achive low latency and high throughput
We can get from [Mishards](../shards/README.md) milvus can be deployed with Mishards as cluster to support massive-scale(10 billion, 100 billion or even larger datasets). However, when it comes to case that datasets is limited(say, millions or tens million) and low latency、high throughput is required, Mishards cannot handle it.
Here, we give a proposal to solve the problem above in production environment based on envoy.
## How to
> Under kubernetes, we can use [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) to handle resources
### Step 1: Docker and Kubernetes environment
If no Docker and Kubernetes environment available, you can build one with [docker](https://v1-17.docs.kubernetes.io/docs/setup/production-environment/container-runtimes/#docker) and [Highly Available Kubernetes clusters](https://v1-17.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/) as reference.
### Step 2: Distributed file system
Milvus cluster relys on [Distributed file system](https://en.wikipedia.org/wiki/Clustered_file_system) to hold data. We can know from [Storge Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/), kubernetes support plenty of plugins to communicate with specific storage engine(such as CephFs、AWS EBS、Azure Disk、GCE PD、Glusterfs ...)
If one Distributed file system is already available, this step can be skipped, otherwise, [GlusterFs](https://www.gluster.org/) can be a option, and we can deploy it following [gluster-kubernetes
](https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md)
After we deployed glusterfs, we can create [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) following [glusterfs StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfs). To be brief, we can create StorageClass from [storageclass.yaml](storage/storageclass.yaml)
```sh
kubectl apply -f storage/storageclass.yaml
```
### Step 3: Namespace
We can create a [Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) for isolation, and all the resource created will be within this namespace(if we need more Milvus cluster, we can create more namespace).
```sh
kubectl apply -f kubernetes/milvus_namespace.yaml
```
What is important to keep in mind is that `StorageClass` created in step 2 can be used across namespaces.
### Step 4: PVC
Based on StorageClass from step 2, we can create [PVC](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) from [milvus_pvc.yaml](kubernetes/milvus_pvc.yaml). Pvcs will be mounted to mysql、envoy、milvus in the following steps.
```sh
kubectl apply -f kubernetes/milvus_pvc.yaml
```
### Step 5. Configmap
[Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) can be used to store all the configurations to be used in the following steps.
```sh
kubectl apply -f kubernetes/configmap/*
```
### Step 6: Mysql
Milvus use mysql to support cluster deployment.
From [Deploying MySQL Server with Docker](https://dev.mysql.com/doc/refman/5.7/en/docker-mysql-more-topics.html) we known, mysql server can deploy with docker and we can create mysql service from [milvus_mysql.yaml](kubernetes/milvus_mysql.yaml).
```sh
kubectl apply -f kubernetes/milvus_mysql.yaml
```
### Step 7: Milvus rw/ro
Following [milvus_rw_servers.yaml](kubernetes/milvus_rw_servers.yaml) and [milvus_ro_servers.yaml](kubernetes/milvus_ro_servers.yaml), we can create milvus rw、ro services.
### Step 8: envoy
**The last but the most.**
[Envoy](https://www.envoyproxy.io/) is an open source edge and service proxy, designed for cloud-native application, we can use envoy to distinguish milvus read/write and get scalability when high throughput is required.
We can create envoy following [envoy_proxy.yaml](kubernetes/envoy_proxy.yaml)
```sh
kubectl apply -f kubernetes/envoy_proxy.yaml
```
Milvus read/write can be distinguished following configuration piece in [envoy-configmap.yaml](kubernetes/configmap/envoy-configmap.yaml) as below:
```yaml
... ...
domains:
- "*"
routes:
- match:
prefix: "/milvus.grpc.MilvusService/Search"
route:
cluster: milvus_backend_ro
timeout: 1s
priority: HIGH
- match:
prefix: "/"
route:
cluster: milvus_backend_rw
timeout: 3600s
priority: HIGH
... ...
```
Milvus can get get scalability following configuration piece in [envoy-configmap.yaml](kubernetes/configmap/envoy-configmap.yaml) and [milvus_ro_servers.yaml](kubernetes/milvus_ro_servers.yaml) as below([Headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) and [Strict DNS](https://www.envoyproxy.io/docs/envoy/v1.11.0/intro/arch_overview/upstream/service_discovery#strict-dns) can help to get scalability when [gRPC](https://grpc.io/blog/grpc-on-http2/#resolvers-and-load-balancers) is used):
`envoy-configmap.yaml`
```yaml
... ...
clusters:
- name: milvus_backend_ro
type: STRICT_DNS
connect_timeout: 1s
lb_policy: ROUND_ROBIN
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
circuit_breakers:
thresholds:
priority: HIGH
max_pending_requests: 20480
max_connections: 20480
max_requests: 20480
max_retries: 1
load_assignment:
cluster_name: milvus_backend_ro
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: milvus-ro-servers
port_value: 19530
protocol: TCP
... ...
```
`milvus_ro_servers.yaml`
```yaml
... ...
kind: Service
apiVersion: v1
metadata:
name: milvus-ro-servers
namespace: milvus-demo
spec:
type: ClusterIP
clusterIP: None
selector:
app: milvus
tier: ro-servers
ports:
- protocol: TCP
port: 19530
targetPort: 19530
name: engine
- protocol: TCP
port: 19121
targetPort: 19121
name: web
... ...
```
Finally, we can access milvus grpc api at port `32000` as configuration piece in [envoy_proxy.yaml](kubernetes/envoy_proxy.yaml) below:
```yaml
apiVersion: v1
kind: Service
metadata:
name: envoy
namespace: milvus-demo
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 32000
selector:
app: milvus
tier: proxy
```
## Extra
### Speed up image pulling
To speed up image pulling, a [private registry](https://goharbor.io/) can be deployed, and we can add extra configuration([specifying-imagepullsecrets-on-a-pod](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod)) to use it.
### Resource assign
[Managing Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) can tells us how memory and cpu usage can be controlled.
### Pod schedule
As described in [assign-pod-node](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node), we can use [NodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) and [Affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) to control where Milvus pod can be located
### Probes
[Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) can be used to detect when Milvus pod is ready, when Milvus pod need to be restarted.
### DaemonSet vs Deployment
[DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) and [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) are two optional ways to deploy `envoy` and `milvus ro` pod.
### helm
[helm](https://helm.sh/) can be used to put all in one just as [milvus-helm](https://github.com/milvus-io/milvus-helm)

138
deploy/kubernetes/configmap/envoy-configmap.yaml Normal file
View File

@ -0,0 +1,138 @@
---
apiVersion: v1
data:
timezone: Asia/shanghai
kind: ConfigMap
metadata:
name: timezone
namespace: milvus-demo
---
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: envoy
name: envoy
namespace: milvus-demo
data:
envoy.yaml: |-
static_resources:
listeners:
- name: milvus_backend_listeners
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 80
filter_chains:
filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
retry_policy:
retry_on: unavailable
domains:
- "*"
routes:
- match:
prefix: "/milvus.grpc.MilvusService/Search"
route:
cluster: milvus_backend_ro
timeout: 1s
priority: HIGH
- match:
prefix: "/"
route:
cluster: milvus_backend_rw
timeout: 3600s
priority: HIGH
http_filters:
- name: envoy.router
add_user_agent: true
access_log:
- name: envoy.file_access_log
filter:
or_filter:
filters:
- status_code_filter:
comparison:
op: GE
value:
default_value: 400
runtime_key: access_log.access_error.status
- duration_filter:
comparison:
op: GE
value:
default_value: 200
runtime_key: access_log.access_error.duration
- traceable_filter: {}
typed_config:
"@type": type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog
path: "/var/log/envoy/access_error.log"
format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% [receivrd-bytes:%BYTES_RECEIVED%] [sent-bytes:%BYTES_SENT%] [time:%DURATION%] %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%REQ(X-LYFT-USER-ID)%\" \"%RESP(GRPC-STATUS)%\"\n"
clusters:
- name: milvus_backend_ro
type: STRICT_DNS
connect_timeout: 1s
lb_policy: ROUND_ROBIN
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
circuit_breakers:
thresholds:
priority: HIGH
max_pending_requests: 20480
max_connections: 20480
max_requests: 20480
max_retries: 1
load_assignment:
cluster_name: milvus_backend_ro
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: milvus-ro-servers
port_value: 19530
protocol: TCP
- name: milvus_backend_rw
type: STRICT_DNS
connect_timeout: 1s
lb_policy: LEAST_REQUEST
dns_lookup_family: V4_ONLY
http2_protocol_options: {}
circuit_breakers:
thresholds:
priority: HIGH
max_pending_requests: 20480
max_connections: 20480
max_requests: 20480
max_retries: 1
load_assignment:
cluster_name: milvus_backend_rw
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: milvus-rw-servers
port_value: 19530
protocol: TCP
cluster_manager:
outlier_detection:
event_log_path: /var/log/envoy/outlier_events.log
admin:
access_log_path: /var/log/envoy/admin_access.log
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901

191
deploy/kubernetes/configmap/milvus-ro-configmap.yaml Normal file
View File

@ -0,0 +1,191 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: milvus-ro-configmap
namespace: milvus-demo
data:
config.yml: |
# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.6
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | If running with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# role | Milvus deployment role: rw / ro | Role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
cluster:
enable: true
role: ro
#----------------------+------------------------------------------------------------+------------+-----------------+
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# timezone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URI | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
general:
timezone: UTC+8
meta_uri: mysql://root:milvusroot@milvus-mysql:3306/{{ template "mysql.database.name" . }}
#----------------------+------------------------------------------------------------+------------+-----------------+
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable HTTP server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus HTTP server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage:
path: /var/milvus/data
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
path: /var/milvus/data/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_size | The size of CPU memory used for caching data for faster | String | 4GB |
# | query. The sum of 'cache_size' and 'insert_buffer_size' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | String | 1GB |
# | The sum of 'insert_buffer_size' and 'cache_size' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_concurrent_insert_request_size | | | |
# | A size limit on the concurrent insert requests to process. | String | 2GB |
# | Milvus can process insert requests from multiple clients | | |
# | concurrently. This setting puts a cap on the memory | | |
# | consumption during this process. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache:
cache_size: 512MB
insert_buffer_size: 512MB
preload_collection: "*"
max_concurrent_insert_request_size: 2GB
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Use GPU devices or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_size | The size of GPU memory per card used for cache. | String | 1GB |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_devices | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_devices | The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu:
enable: false
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range | String | 1024MB |
# | [512MB, 4096MB]. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_rotate_num | The maximum number of log files that Milvus keeps for each | Integer | 0 |
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_to_stdout | Whether logging to standard output. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_to_file | Whether logging to log files. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: info
trace.enable: true
path: /var/milvus/logs
max_log_file_size: 1024MB
log_rotate_num: 3
log_to_stdout: false
log_to_file: true
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091

191
deploy/kubernetes/configmap/milvus-rw-configmap.yaml Normal file
View File

@ -0,0 +1,191 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: milvus-rw-configmap
namespace: milvus-demo
data:
config.yml: |
# Copyright (C) 2019-2020 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.
version: 0.6
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cluster Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | If running with Mishards, set true, otherwise false. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# role | Milvus deployment role: rw / ro | Role | rw |
#----------------------+------------------------------------------------------------+------------+-----------------+
cluster:
enable: true
role: rw
#----------------------+------------------------------------------------------------+------------+-----------------+
# General Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# timezone | Use UTC-x or UTC+x to specify a time zone. | Timezone | UTC+8 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# meta_uri | URI for metadata storage, using SQLite (for single server | URI | sqlite://:@:/ |
# | Milvus) or MySQL (for distributed cluster Milvus). | | |
# | Format: dialect://username:password@host:port/database | | |
# | Keep 'dialect://:@:/', 'dialect' can be either 'sqlite' or | | |
# | 'mysql', replace other texts with real values. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
general:
timezone: UTC+8
meta_uri: mysql://root:milvusroot@milvus-mysql:3306/{{ template "mysql.database.name" . }}
#----------------------+------------------------------------------------------------+------------+-----------------+
# Network Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.address | IP address that Milvus server monitors. | IP | 0.0.0.0 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# bind.port | Port that Milvus server monitors. Port range (1024, 65535) | Integer | 19530 |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.enable | Enable HTTP server or not. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# http.port | Port that Milvus HTTP server monitors. | Integer | 19121 |
# | Port range (1024, 65535) | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
network:
bind.address: 0.0.0.0
bind.port: 19530
http.enable: true
http.port: 19121
#----------------------+------------------------------------------------------------+------------+-----------------+
# Storage Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Path used to save meta data, vector data and index data. | Path | /var/lib/milvus |
#----------------------+------------------------------------------------------------+------------+-----------------+
# auto_flush_interval | The interval, in seconds, at which Milvus automatically | Integer | 1 (s) |
# | flushes data to disk. | | |
# | 0 means disable the regular flush. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
storage:
path: /var/milvus/data
auto_flush_interval: 1
#----------------------+------------------------------------------------------------+------------+-----------------+
# WAL Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Whether to enable write-ahead logging (WAL) in Milvus. | Boolean | true |
# | If WAL is enabled, Milvus writes all data changes to log | | |
# | files in advance before implementing data changes. WAL | | |
# | ensures the atomicity and durability for Milvus operations.| | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Location of WAL log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
wal:
enable: true
path: /var/milvus/data/wal
#----------------------+------------------------------------------------------------+------------+-----------------+
# Cache Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_size | The size of CPU memory used for caching data for faster | String | 4GB |
# | query. The sum of 'cache_size' and 'insert_buffer_size' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# insert_buffer_size | Buffer size used for data insertion. | String | 1GB |
# | The sum of 'insert_buffer_size' and 'cache_size' | | |
# | must be less than system memory size. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# preload_collection | A comma-separated list of collection names that need to | StringList | |
# | be pre-loaded when Milvus server starts up. | | |
# | '*' means preload all existing tables (single-quote or | | |
# | double-quote required). | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_concurrent_insert_request_size | | | |
# | A size limit on the concurrent insert requests to process. | String | 2GB |
# | Milvus can process insert requests from multiple clients | | |
# | concurrently. This setting puts a cap on the memory | | |
# | consumption during this process. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
cache:
cache_size: 512MB
insert_buffer_size: 512MB
preload_collection: "*"
max_concurrent_insert_request_size: 2GB
#----------------------+------------------------------------------------------------+------------+-----------------+
# GPU Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Use GPU devices or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# cache_size | The size of GPU memory per card used for cache. | String | 1GB |
#----------------------+------------------------------------------------------------+------------+-----------------+
# gpu_search_threshold | A Milvus performance tuning parameter. This value will be | Integer | 1000 |
# | compared with 'nq' to decide if the search computation will| | |
# | be executed on GPUs only. | | |
# | If nq >= gpu_search_threshold, the search computation will | | |
# | be executed on GPUs only; | | |
# | if nq < gpu_search_threshold, the search computation will | | |
# | be executed on both CPUs and GPUs. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# search_devices | The list of GPU devices used for search computation. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# build_index_devices | The list of GPU devices used for index building. | DeviceList | gpu0 |
# | Must be in format gpux. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
gpu:
enable: false
cache_size: 1GB
gpu_search_threshold: 1000
search_devices:
- gpu0
build_index_devices:
- gpu0
#----------------------+------------------------------------------------------------+------------+-----------------+
# Logs Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# level | Log level in Milvus. Must be one of debug, info, warning, | String | debug |
# | error, fatal | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# trace.enable | Whether to enable trace level logging in Milvus. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
# path | Absolute path to the folder holding the log files. | String | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# max_log_file_size | The maximum size of each log file, size range | String | 1024MB |
# | [512MB, 4096MB]. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_rotate_num | The maximum number of log files that Milvus keeps for each | Integer | 0 |
# | logging level, num range [0, 1024], 0 means unlimited. | | |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_to_stdout | Whether logging to standard output. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# log_to_file | Whether logging to log files. | Boolean | true |
#----------------------+------------------------------------------------------------+------------+-----------------+
logs:
level: info
trace.enable: true
path: /var/milvus/logs
max_log_file_size: 1024MB
log_rotate_num: 3
log_to_stdout: false
log_to_file: true
#----------------------+------------------------------------------------------------+------------+-----------------+
# Metric Config | Description | Type | Default |
#----------------------+------------------------------------------------------------+------------+-----------------+
# enable | Enable monitoring function or not. | Boolean | false |
#----------------------+------------------------------------------------------------+------------+-----------------+
# address | Pushgateway address | IP | 127.0.0.1 +
#----------------------+------------------------------------------------------------+------------+-----------------+
# port | Pushgateway port, port range (1024, 65535) | Integer | 9091 |
#----------------------+------------------------------------------------------------+------------+-----------------+
metric:
enable: false
address: 127.0.0.1
port: 9091

22
deploy/kubernetes/configmap/mysql_configmap.yaml Normal file
View File

@ -0,0 +1,22 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: milvus-mysql-configmap
namespace: milvus-demo
data:
milvus_mysql_config.yml: |
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /data
log-error = /var/log/mysql/error.log # mount out to host
# By default we only accept connections from localhost
bind-address = 0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
skip-character-set-client-handshake = true
max_connections = 1000
wait_timeout = 31536000

147
deploy/kubernetes/envoy_proxy.yaml Normal file
View File

@ -0,0 +1,147 @@
apiVersion: v1
kind: Service
metadata:
name: envoy
namespace: milvus-demo
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 32000
selector:
app: milvus
tier: proxy
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: milvus
tier: proxy
name: milvus-proxy
namespace: milvus-demo
spec:
selector:
matchLabels:
app: milvus
tier: proxy
template:
metadata:
labels:
app: milvus
tier: proxy
spec:
containers:
- image: envoyproxy/envoy:v1.11.0
imagePullPolicy: IfNotPresent
name: milvus-proxy
ports:
- containerPort: 80
name: tcp
protocol: TCP
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/localtime
name: time
- mountPath: /etc/timezone
name: timezone
subPath: timezone
- mountPath: /etc/envoy/envoy.yaml
name: envoy
subPath: envoy.yaml
- mountPath: /var/log/envoy
name: milvus-log-disk
subPath: envoylog
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes:
- hostPath:
path: /etc/localtime
type: ""
name: time
- configMap:
defaultMode: 420
name: timezone
name: timezone
- configMap:
defaultMode: 420
name: envoy
name: envoy
- configMap:
defaultMode: 420
name: milvus-proxy-configmap
name: milvus-proxy-configmap
- name: milvus-log-disk
persistentVolumeClaim:
claimName: milvus-proxy-log
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: milvus
tier: proxy
name: milvus-proxy
namespace: milvus-demo
spec:
replicas: 0
selector:
matchLabels:
app: milvus
tier: proxy
template:
metadata:
labels:
app: milvus
tier: proxy
spec:
containers:
- image: docker.ke.com:80/vsearch/envoyproxy/envoy:v1.11.0
imagePullPolicy: IfNotPresent
name: milvus-proxy
ports:
- containerPort: 80
name: tcp
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/localtime
name: time
- mountPath: /etc/timezone
name: timezone
subPath: timezone
- mountPath: /etc/envoy/envoy.yaml
name: envoy
subPath: envoy.yaml
- mountPath: /var/log/envoy
name: milvus-log-disk
subPath: envoylog
restartPolicy: Always
volumes:
- hostPath:
path: /etc/localtime
type: ""
name: time
- configMap:
defaultMode: 420
name: timezone
name: timezone
- configMap:
defaultMode: 420
name: envoy
name: envoy
- configMap:
defaultMode: 420
name: milvus-proxy-configmap
name: milvus-proxy-configmap
- name: milvus-log-disk
persistentVolumeClaim:
claimName: milvus-proxy-log

75
deploy/kubernetes/milvus_mysql.yaml Normal file
View File

@ -0,0 +1,75 @@
kind: Service
apiVersion: v1
metadata:
name: milvus-mysql
namespace: milvus-demo
spec:
type: ClusterIP
selector:
app: milvus
tier: mysql
ports:
- protocol: TCP
port: 3306
targetPort: 3306
name: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: milvus-mysql
namespace: milvus-demo
spec:
selector:
matchLabels:
app: milvus
tier: mysql
replicas: 1
template:
metadata:
labels:
app: milvus
tier: mysql
spec:
containers:
- name: milvus-mysql
image: mysql:5.7
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- "/bin/sh"
- "-c"
- |
echo 'start init schema' >> ~/start.log
mysql -h milvus-mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "CREATE DATABASE IF NOT EXISTS ${MYSQL_DATABASE};" >> ~/start.log
mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "GRANT ALL PRIVILEGES ON ${MYSQL_DATABASE}.* TO 'root'@'%';" >> ~/start.log
echo 'end int schema' >> ~/start.log
env:
- name: MYSQL_ROOT_PASSWORD
value: milvusroot
- name: MYSQL_DATABASE
value: milvus-demo
ports:
- name: mysql-port
containerPort: 3306
volumeMounts:
- name: milvus-mysql-disk
mountPath: /data
subPath: mysql
- name: milvus-mysql-configmap
mountPath: /etc/mysql/mysql.conf.d/mysqld.cnf
subPath: milvus_mysql_config.yml
volumes:
- name: milvus-mysql-disk
persistentVolumeClaim:
claimName: milvus-mysql-disk
- name: milvus-mysql-configmap
configMap:
name: milvus-mysql-configmap

4
deploy/kubernetes/milvus_namespace.yaml Normal file
View File

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: milvus-demo

46
deploy/kubernetes/milvus_pvc.yaml Normal file
View File

@ -0,0 +1,46 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: milvus-db-disk
namespace: milvus-demo
annotations:
volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: milvus-mysql-disk
namespace: milvus-demo
annotations:
volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: milvus-proxy-log
namespace: milvus-demo
annotations:
volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi

156
deploy/kubernetes/milvus_ro_servers.yaml Normal file
View File

@ -0,0 +1,156 @@
kind: Service
apiVersion: v1
metadata:
name: milvus-ro-servers
namespace: milvus-demo
spec:
type: ClusterIP
clusterIP: None
selector:
app: milvus
tier: ro-servers
ports:
- protocol: TCP
port: 19530
targetPort: 19530
name: engine
- protocol: TCP
port: 19121
targetPort: 19121
name: web
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: milvus-ro-servers
namespace: milvus-demo
spec:
selector:
matchLabels:
app: milvus
tier: ro-servers
template:
metadata:
labels:
app: milvus
tier: ro-servers
spec:
containers:
- image: milvusdb/milvus:0.11.0-cpu-d101620-4c44c0
imagePullPolicy: IfNotPresent
livenessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
name: milvus-ro-server
ports:
- containerPort: 19530
name: engine
protocol: TCP
- containerPort: 19121
name: web
protocol: TCP
readinessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 10
failureThreshold: 3
volumeMounts:
- name: milvus-db-disk
mountPath: /var/milvus/data
subPath: data
- name: milvus-db-disk
mountPath: /var/milvus/logs
subPath: logs
- name: milvus-ro-configmap
mountPath: /var/lib/milvus/conf/server_config.yaml
subPath: config.yml
volumes:
- name: milvus-ro-configmap
configMap:
name: milvus-ro-configmap
- name: milvus-db-disk
persistentVolumeClaim:
claimName: milvus-db-disk
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: milvus-ro-servers
namespace: milvus-demo
spec:
replicas: 0
selector:
matchLabels:
app: milvus
tier: ro-servers
template:
metadata:
labels:
app: milvus
tier: ro-servers
spec:
terminationGracePeriodSeconds: 11
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: tier
operator: In
values:
- ro-servers
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- image: milvusdb/milvus:0.11.0-cpu-d101620-4c44c0
imagePullPolicy: IfNotPresent
livenessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
name: milvus-ro-server
ports:
- containerPort: 19530
name: engine
protocol: TCP
- containerPort: 19121
name: web
protocol: TCP
readinessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 10
failureThreshold: 3
volumeMounts:
- name: milvus-db-disk
mountPath: /var/milvus/data
subPath: data
- name: milvus-db-disk
mountPath: /var/milvus/logs
subPath: logs
- name: milvus-ro-configmap
mountPath: /var/lib/milvus/conf/milvus.yaml
subPath: config.yml
volumes:
- name: milvus-ro-configmap
configMap:
name: milvus-ro-configmap
- name: milvus-db-disk
persistentVolumeClaim:
claimName: milvus-db-disk

84
deploy/kubernetes/milvus_rw_servers.yaml Normal file
View File

@ -0,0 +1,84 @@
kind: Service
apiVersion: v1
metadata:
name: milvus-rw-servers
namespace: milvus-demo
spec:
type: ClusterIP
clusterIP: None
selector:
app: milvus
tier: rw-servers
ports:
- protocol: TCP
port: 19530
targetPort: 19530
name: engine
- protocol: TCP
port: 19121
targetPort: 19121
name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: milvus-rw-servers
namespace: milvus-demo
spec:
selector:
matchLabels:
app: milvus
tier: rw-servers
strategy:
type: Recreate
replicas: 1
template:
metadata:
labels:
app: milvus
tier: rw-servers
spec:
containers:
- image: docker.ke.com:80/vsearch/milvus/milvus:0.11.0-cpu-d101620-4c44c0
imagePullPolicy: IfNotPresent
livenessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
name: milvus-rw-server
ports:
- containerPort: 19530
name: engine
protocol: TCP
- containerPort: 19121
name: web
protocol: TCP
readinessProbe:
tcpSocket:
port: 19530
initialDelaySeconds: 15
periodSeconds: 15
timeoutSeconds: 10
failureThreshold: 3
volumeMounts:
- name: milvus-db-disk
mountPath: /var/milvus/data
subPath: data
- name: milvus-db-disk
mountPath: /var/milvus/logs
subPath: logs
- name: milvus-rw-configmap
mountPath: /var/lib/milvus/conf/milvus.yaml
subPath: config.yml
volumes:
- name: milvus-rw-configmap
configMap:
name: milvus-rw-configmap
- name: milvus-db-disk
persistentVolumeClaim:
claimName: milvus-db-disk

24
deploy/storage/storageclass.yaml Normal file
View File

@ -0,0 +1,24 @@
# create if absent
# apiVersion: v1
# kind: Namespace
# metadata:
# name: glusterfs
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: gluster-heketi
namespace: glusterfs
provisioner: kubernetes.io/glusterfs
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
# resturl: "ip:port"
# restuser: "admin"
# restuserkey: "admin"
volumetype: "replicate:3"