mirror of https://github.com/milvus-io/milvus.git
Merge pull request #87 from milvus-io/master
Merge from master Former-commit-id: df85d7d6895b385a9e367609872fc86edc379176pull/191/head
Jin Hai
GitHub
commit
320e5af8f2
|
|
@ -11,7 +11,7 @@ Please mark all change in change log and use the ticket from JIRA.
|
|||
## Feature
|
||||
## Task
|
||||
|
||||
# Milvus 0.5.0 (TODO)
|
||||
# Milvus 0.5.0 (2019-10-21)
|
||||
|
||||
## Bug
|
||||
- MS-568 - Fix gpuresource free error
|
||||
|
|
|
|||
49
README.md
49
README.md
|
|
@ -1,7 +1,9 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
[](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)
|
||||
|
||||
- [Slack Community](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
|
||||
- [Twitter](https://twitter.com/milvusio)
|
||||
|
|
@ -13,34 +15,49 @@
|
|||
|
||||
# Welcome to Milvus
|
||||
|
||||
Firstly, welcome, and thanks for your interest in [Milvus](https://milvus.io)! No matter who you are, what you do, we greatly appreciate your contribution to help us reinvent data science with Milvus. :beers:
|
||||
|
||||
## What is Milvus
|
||||
|
||||
Milvus is an open source vector search engine which provides state-of-the-art similarity search and analysis for billion-scale feature vectors.
|
||||
Milvus is an open source similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
|
||||
|
||||
Milvus provides stable Python, Java and C++ APIs.
|
||||
|
||||
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/Releases/v0.4.0/).
|
||||
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/Releases/v0.5.0/).
|
||||
|
||||
- GPU-accelerated search engine
|
||||
- Heterogeneous computing
|
||||
|
||||
Milvus uses CPU/GPU heterogeneous computing architecture to process feature vectors, and are orders of magnitudes faster than traditional databases.
|
||||
Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency.
|
||||
|
||||
- Various indexes
|
||||
- Multiple indexes
|
||||
|
||||
Milvus supports quantization indexing, tree-based indexing, and graph indexing algorithms.
|
||||
Milvus supports a variety of indexing types that employs quantization, tree-based, and graph indexing techniques.
|
||||
|
||||
- Intelligent scheduling
|
||||
- Intelligent resource management
|
||||
|
||||
Milvus optimizes the search computation and index building according to your data size and available resources.
|
||||
Milvus automatically adapts search computation and index building processes based on your datasets and available resources.
|
||||
|
||||
- Horizontal scalability
|
||||
|
||||
Milvus expands computation and storage by adding nodes during runtime, which allows you to scale the data size without redesigning the system.
|
||||
Milvus supports online / offline expansion to scale both storage and computation resources with simple commands.
|
||||
|
||||
- High availability
|
||||
|
||||
Milvus is integrated with Kubernetes framework so that all single point of failures could be avoided.
|
||||
|
||||
- High compatibility
|
||||
|
||||
Milvus is compatible with almost all deep learning models and major programming languages such as Python, Java and C++, etc.
|
||||
|
||||
- Ease of use
|
||||
|
||||
Milvus can be easily installed in a few steps and enables you to exclusively focus on feature vectors.
|
||||
|
||||
- Visualized monitor
|
||||
|
||||
You can track system performance on Prometheus-based GUI monitor dashboards.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
## Get started
|
||||
|
||||
|
|
@ -117,20 +134,20 @@ To edit Milvus settings in `conf/server_config.yaml` and `conf/log_config.conf`,
|
|||
|
||||
#### Run Python example code
|
||||
|
||||
Make sure [Python 3.4](https://www.python.org/downloads/) or higher is already installed and in use.
|
||||
Make sure [Python 3.5](https://www.python.org/downloads/) or higher is already installed and in use.
|
||||
|
||||
Install Milvus Python SDK.
|
||||
|
||||
```shell
|
||||
# Install Milvus Python SDK
|
||||
$ pip install pymilvus==0.2.0
|
||||
$ pip install pymilvus==0.2.3
|
||||
```
|
||||
|
||||
Create a new file `example.py`, and add [Python example code](https://github.com/milvus-io/pymilvus/blob/master/examples/AdvancedExample.py) to it.
|
||||
|
||||
Run the example code.
|
||||
|
||||
```python
|
||||
```shell
|
||||
# Run Milvus Python example
|
||||
$ python3 example.py
|
||||
```
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
#Configuration File for CodeCov
|
||||
coverage:
|
||||
precision: 2
|
||||
round: down
|
||||
range: "70...100"
|
||||
|
||||
status:
|
||||
project: on
|
||||
patch: yes
|
||||
changes: no
|
||||
|
||||
comment:
|
||||
layout: "header, diff, changes, tree"
|
||||
behavior: default
|
||||
|
|
@ -307,71 +307,71 @@ XSearchTask::MergeTopkToResultSet(const std::vector<int64_t>& input_ids, const s
|
|||
}
|
||||
}
|
||||
|
||||
void
|
||||
XSearchTask::MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,
|
||||
const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance,
|
||||
uint64_t src_input_k, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
if (src_ids.empty() || src_distance.empty()) {
|
||||
return;
|
||||
}
|
||||
|
||||
uint64_t output_k = std::min(topk, tar_input_k + src_input_k);
|
||||
std::vector<int64_t> id_buf(nq * output_k, -1);
|
||||
std::vector<float> dist_buf(nq * output_k, 0.0);
|
||||
|
||||
uint64_t buf_k, src_k, tar_k;
|
||||
uint64_t src_idx, tar_idx, buf_idx;
|
||||
uint64_t src_input_k_multi_i, tar_input_k_multi_i, buf_k_multi_i;
|
||||
|
||||
for (uint64_t i = 0; i < nq; i++) {
|
||||
src_input_k_multi_i = src_input_k * i;
|
||||
tar_input_k_multi_i = tar_input_k * i;
|
||||
buf_k_multi_i = output_k * i;
|
||||
buf_k = src_k = tar_k = 0;
|
||||
while (buf_k < output_k && src_k < src_input_k && tar_k < tar_input_k) {
|
||||
src_idx = src_input_k_multi_i + src_k;
|
||||
tar_idx = tar_input_k_multi_i + tar_k;
|
||||
buf_idx = buf_k_multi_i + buf_k;
|
||||
if ((ascending && src_distance[src_idx] < tar_distance[tar_idx]) ||
|
||||
(!ascending && src_distance[src_idx] > tar_distance[tar_idx])) {
|
||||
id_buf[buf_idx] = src_ids[src_idx];
|
||||
dist_buf[buf_idx] = src_distance[src_idx];
|
||||
src_k++;
|
||||
} else {
|
||||
id_buf[buf_idx] = tar_ids[tar_idx];
|
||||
dist_buf[buf_idx] = tar_distance[tar_idx];
|
||||
tar_k++;
|
||||
}
|
||||
buf_k++;
|
||||
}
|
||||
|
||||
if (buf_k < output_k) {
|
||||
if (src_k < src_input_k) {
|
||||
while (buf_k < output_k && src_k < src_input_k) {
|
||||
src_idx = src_input_k_multi_i + src_k;
|
||||
buf_idx = buf_k_multi_i + buf_k;
|
||||
id_buf[buf_idx] = src_ids[src_idx];
|
||||
dist_buf[buf_idx] = src_distance[src_idx];
|
||||
src_k++;
|
||||
buf_k++;
|
||||
}
|
||||
} else {
|
||||
while (buf_k < output_k && tar_k < tar_input_k) {
|
||||
tar_idx = tar_input_k_multi_i + tar_k;
|
||||
buf_idx = buf_k_multi_i + buf_k;
|
||||
id_buf[buf_idx] = tar_ids[tar_idx];
|
||||
dist_buf[buf_idx] = tar_distance[tar_idx];
|
||||
tar_k++;
|
||||
buf_k++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tar_ids.swap(id_buf);
|
||||
tar_distance.swap(dist_buf);
|
||||
tar_input_k = output_k;
|
||||
}
|
||||
// void
|
||||
// XSearchTask::MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,
|
||||
// const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance,
|
||||
// uint64_t src_input_k, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
// if (src_ids.empty() || src_distance.empty()) {
|
||||
// return;
|
||||
// }
|
||||
//
|
||||
// uint64_t output_k = std::min(topk, tar_input_k + src_input_k);
|
||||
// std::vector<int64_t> id_buf(nq * output_k, -1);
|
||||
// std::vector<float> dist_buf(nq * output_k, 0.0);
|
||||
//
|
||||
// uint64_t buf_k, src_k, tar_k;
|
||||
// uint64_t src_idx, tar_idx, buf_idx;
|
||||
// uint64_t src_input_k_multi_i, tar_input_k_multi_i, buf_k_multi_i;
|
||||
//
|
||||
// for (uint64_t i = 0; i < nq; i++) {
|
||||
// src_input_k_multi_i = src_input_k * i;
|
||||
// tar_input_k_multi_i = tar_input_k * i;
|
||||
// buf_k_multi_i = output_k * i;
|
||||
// buf_k = src_k = tar_k = 0;
|
||||
// while (buf_k < output_k && src_k < src_input_k && tar_k < tar_input_k) {
|
||||
// src_idx = src_input_k_multi_i + src_k;
|
||||
// tar_idx = tar_input_k_multi_i + tar_k;
|
||||
// buf_idx = buf_k_multi_i + buf_k;
|
||||
// if ((ascending && src_distance[src_idx] < tar_distance[tar_idx]) ||
|
||||
// (!ascending && src_distance[src_idx] > tar_distance[tar_idx])) {
|
||||
// id_buf[buf_idx] = src_ids[src_idx];
|
||||
// dist_buf[buf_idx] = src_distance[src_idx];
|
||||
// src_k++;
|
||||
// } else {
|
||||
// id_buf[buf_idx] = tar_ids[tar_idx];
|
||||
// dist_buf[buf_idx] = tar_distance[tar_idx];
|
||||
// tar_k++;
|
||||
// }
|
||||
// buf_k++;
|
||||
// }
|
||||
//
|
||||
// if (buf_k < output_k) {
|
||||
// if (src_k < src_input_k) {
|
||||
// while (buf_k < output_k && src_k < src_input_k) {
|
||||
// src_idx = src_input_k_multi_i + src_k;
|
||||
// buf_idx = buf_k_multi_i + buf_k;
|
||||
// id_buf[buf_idx] = src_ids[src_idx];
|
||||
// dist_buf[buf_idx] = src_distance[src_idx];
|
||||
// src_k++;
|
||||
// buf_k++;
|
||||
// }
|
||||
// } else {
|
||||
// while (buf_k < output_k && tar_k < tar_input_k) {
|
||||
// tar_idx = tar_input_k_multi_i + tar_k;
|
||||
// buf_idx = buf_k_multi_i + buf_k;
|
||||
// id_buf[buf_idx] = tar_ids[tar_idx];
|
||||
// dist_buf[buf_idx] = tar_distance[tar_idx];
|
||||
// tar_k++;
|
||||
// buf_k++;
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// tar_ids.swap(id_buf);
|
||||
// tar_distance.swap(dist_buf);
|
||||
// tar_input_k = output_k;
|
||||
//}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
|
|
|
|||
|
|
@ -42,10 +42,10 @@ class XSearchTask : public Task {
|
|||
MergeTopkToResultSet(const std::vector<int64_t>& input_ids, const std::vector<float>& input_distance,
|
||||
uint64_t input_k, uint64_t nq, uint64_t topk, bool ascending, scheduler::ResultSet& result);
|
||||
|
||||
static void
|
||||
MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,
|
||||
const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance, uint64_t src_input_k,
|
||||
uint64_t nq, uint64_t topk, bool ascending);
|
||||
// static void
|
||||
// MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,
|
||||
// const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance, uint64_t
|
||||
// src_input_k, uint64_t nq, uint64_t topk, bool ascending);
|
||||
|
||||
public:
|
||||
TableFileSchemaPtr file_;
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ namespace ms = milvus::scheduler;
|
|||
void
|
||||
BuildResult(std::vector<int64_t>& output_ids,
|
||||
std::vector<float>& output_distance,
|
||||
uint64_t input_k,
|
||||
uint64_t topk,
|
||||
uint64_t nq,
|
||||
bool ascending) {
|
||||
|
|
@ -39,9 +40,15 @@ BuildResult(std::vector<int64_t>& output_ids,
|
|||
output_distance.resize(nq * topk);
|
||||
|
||||
for (uint64_t i = 0; i < nq; i++) {
|
||||
for (uint64_t j = 0; j < topk; j++) {
|
||||
//insert valid items
|
||||
for (uint64_t j = 0; j < input_k; j++) {
|
||||
output_ids[i * topk + j] = (int64_t)(drand48() * 100000);
|
||||
output_distance[i * topk + j] = ascending ? (j + drand48()) : ((topk - j) + drand48());
|
||||
output_distance[i * topk + j] = ascending ? (j + drand48()) : ((input_k - j) + drand48());
|
||||
}
|
||||
//insert invalid items
|
||||
for (uint64_t j = input_k; j < topk; j++) {
|
||||
output_ids[i * topk + j] = -1;
|
||||
output_distance[i * topk + j] = -1.0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -83,23 +90,32 @@ CheckTopkResult(const std::vector<int64_t>& input_ids_1,
|
|||
ASSERT_EQ(input_ids_1.size(), input_distance_1.size());
|
||||
ASSERT_EQ(input_ids_2.size(), input_distance_2.size());
|
||||
|
||||
uint64_t input_k1 = input_ids_1.size() / nq;
|
||||
uint64_t input_k2 = input_ids_2.size() / nq;
|
||||
|
||||
for (int64_t i = 0; i < nq; i++) {
|
||||
std::vector<float>
|
||||
src_vec(input_distance_1.begin() + i * input_k1, input_distance_1.begin() + (i + 1) * input_k1);
|
||||
src_vec(input_distance_1.begin() + i * topk, input_distance_1.begin() + (i + 1) * topk);
|
||||
src_vec.insert(src_vec.end(),
|
||||
input_distance_2.begin() + i * input_k2,
|
||||
input_distance_2.begin() + (i + 1) * input_k2);
|
||||
input_distance_2.begin() + i * topk,
|
||||
input_distance_2.begin() + (i + 1) * topk);
|
||||
if (ascending) {
|
||||
std::sort(src_vec.begin(), src_vec.end());
|
||||
} else {
|
||||
std::sort(src_vec.begin(), src_vec.end(), std::greater<float>());
|
||||
}
|
||||
|
||||
uint64_t n = std::min(topk, input_k1 + input_k2);
|
||||
//erase invalid items
|
||||
std::vector<float>::iterator iter;
|
||||
for (iter = src_vec.begin(); iter != src_vec.end();) {
|
||||
if (*iter < 0.0)
|
||||
iter = src_vec.erase(iter);
|
||||
else
|
||||
++iter;
|
||||
}
|
||||
|
||||
uint64_t n = std::min(topk, result[i].size());
|
||||
for (uint64_t j = 0; j < n; j++) {
|
||||
if (result[i][j].first < 0) {
|
||||
continue;
|
||||
}
|
||||
if (src_vec[j] != result[i][j].second) {
|
||||
std::cout << src_vec[j] << " " << result[i][j].second << std::endl;
|
||||
}
|
||||
|
|
@ -110,12 +126,13 @@ CheckTopkResult(const std::vector<int64_t>& input_ids_1,
|
|||
|
||||
} // namespace
|
||||
|
||||
void MergeTopkToResultSetTest(uint64_t topk_1, uint64_t topk_2, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
void
|
||||
MergeTopkToResultSetTest(uint64_t topk_1, uint64_t topk_2, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
std::vector<int64_t> ids1, ids2;
|
||||
std::vector<float> dist1, dist2;
|
||||
ms::ResultSet result;
|
||||
BuildResult(ids1, dist1, topk_1, nq, ascending);
|
||||
BuildResult(ids2, dist2, topk_2, nq, ascending);
|
||||
BuildResult(ids1, dist1, topk_1, topk, nq, ascending);
|
||||
BuildResult(ids2, dist2, topk_2, topk, nq, ascending);
|
||||
ms::XSearchTask::MergeTopkToResultSet(ids1, dist1, topk_1, nq, topk, ascending, result);
|
||||
ms::XSearchTask::MergeTopkToResultSet(ids2, dist2, topk_2, nq, topk, ascending, result);
|
||||
CheckTopkResult(ids1, dist1, ids2, dist2, topk, nq, ascending, result);
|
||||
|
|
@ -134,70 +151,72 @@ TEST(DBSearchTest, MERGE_RESULT_SET_TEST) {
|
|||
MergeTopkToResultSetTest(TOP_K, TOP_K, NQ, TOP_K, false);
|
||||
|
||||
/* test3, id1/dist1 small topk */
|
||||
MergeTopkToResultSetTest(TOP_K/2, TOP_K, NQ, TOP_K, true);
|
||||
MergeTopkToResultSetTest(TOP_K/2, TOP_K, NQ, TOP_K, false);
|
||||
MergeTopkToResultSetTest(TOP_K / 2, TOP_K, NQ, TOP_K, true);
|
||||
MergeTopkToResultSetTest(TOP_K / 2, TOP_K, NQ, TOP_K, false);
|
||||
|
||||
/* test4, id1/dist1 small topk, id2/dist2 small topk */
|
||||
MergeTopkToResultSetTest(TOP_K/2, TOP_K/3, NQ, TOP_K, true);
|
||||
MergeTopkToResultSetTest(TOP_K/2, TOP_K/3, NQ, TOP_K, false);
|
||||
MergeTopkToResultSetTest(TOP_K / 2, TOP_K / 3, NQ, TOP_K, true);
|
||||
MergeTopkToResultSetTest(TOP_K / 2, TOP_K / 3, NQ, TOP_K, false);
|
||||
}
|
||||
|
||||
void MergeTopkArrayTest(uint64_t topk_1, uint64_t topk_2, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
std::vector<int64_t> ids1, ids2;
|
||||
std::vector<float> dist1, dist2;
|
||||
ms::ResultSet result;
|
||||
BuildResult(ids1, dist1, topk_1, nq, ascending);
|
||||
BuildResult(ids2, dist2, topk_2, nq, ascending);
|
||||
uint64_t result_topk = std::min(topk, topk_1 + topk_2);
|
||||
ms::XSearchTask::MergeTopkArray(ids1, dist1, topk_1, ids2, dist2, topk_2, nq, topk, ascending);
|
||||
if (ids1.size() != result_topk * nq) {
|
||||
std::cout << ids1.size() << " " << result_topk * nq << std::endl;
|
||||
}
|
||||
ASSERT_TRUE(ids1.size() == result_topk * nq);
|
||||
ASSERT_TRUE(dist1.size() == result_topk * nq);
|
||||
for (uint64_t i = 0; i < nq; i++) {
|
||||
for (uint64_t k = 1; k < result_topk; k++) {
|
||||
if (ascending) {
|
||||
if (dist1[i * result_topk + k] < dist1[i * result_topk + k - 1]) {
|
||||
std::cout << dist1[i * result_topk + k - 1] << " " << dist1[i * result_topk + k] << std::endl;
|
||||
}
|
||||
ASSERT_TRUE(dist1[i * result_topk + k] >= dist1[i * result_topk + k - 1]);
|
||||
} else {
|
||||
if (dist1[i * result_topk + k] > dist1[i * result_topk + k - 1]) {
|
||||
std::cout << dist1[i * result_topk + k - 1] << " " << dist1[i * result_topk + k] << std::endl;
|
||||
}
|
||||
ASSERT_TRUE(dist1[i * result_topk + k] <= dist1[i * result_topk + k - 1]);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
//void MergeTopkArrayTest(uint64_t topk_1, uint64_t topk_2, uint64_t nq, uint64_t topk, bool ascending) {
|
||||
// std::vector<int64_t> ids1, ids2;
|
||||
// std::vector<float> dist1, dist2;
|
||||
// ms::ResultSet result;
|
||||
// BuildResult(ids1, dist1, topk_1, topk, nq, ascending);
|
||||
// BuildResult(ids2, dist2, topk_2, topk, nq, ascending);
|
||||
// uint64_t result_topk = std::min(topk, topk_1 + topk_2);
|
||||
// ms::XSearchTask::MergeTopkArray(ids1, dist1, topk_1, ids2, dist2, topk_2, nq, topk, ascending);
|
||||
// if (ids1.size() != result_topk * nq) {
|
||||
// std::cout << ids1.size() << " " << result_topk * nq << std::endl;
|
||||
// }
|
||||
// ASSERT_TRUE(ids1.size() == result_topk * nq);
|
||||
// ASSERT_TRUE(dist1.size() == result_topk * nq);
|
||||
// for (uint64_t i = 0; i < nq; i++) {
|
||||
// for (uint64_t k = 1; k < result_topk; k++) {
|
||||
// float f0 = dist1[i * topk + k - 1];
|
||||
// float f1 = dist1[i * topk + k];
|
||||
// if (ascending) {
|
||||
// if (f1 < f0) {
|
||||
// std::cout << f0 << " " << f1 << std::endl;
|
||||
// }
|
||||
// ASSERT_TRUE(f1 >= f0);
|
||||
// } else {
|
||||
// if (f1 > f0) {
|
||||
// std::cout << f0 << " " << f1 << std::endl;
|
||||
// }
|
||||
// ASSERT_TRUE(f1 <= f0);
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
//}
|
||||
|
||||
TEST(DBSearchTest, MERGE_ARRAY_TEST) {
|
||||
uint64_t NQ = 15;
|
||||
uint64_t TOP_K = 64;
|
||||
//TEST(DBSearchTest, MERGE_ARRAY_TEST) {
|
||||
// uint64_t NQ = 15;
|
||||
// uint64_t TOP_K = 64;
|
||||
//
|
||||
// /* test1, id1/dist1 valid, id2/dist2 empty */
|
||||
// MergeTopkArrayTest(TOP_K, 0, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K, 0, NQ, TOP_K, false);
|
||||
// MergeTopkArrayTest(0, TOP_K, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(0, TOP_K, NQ, TOP_K, false);
|
||||
|
||||
/* test1, id1/dist1 valid, id2/dist2 empty */
|
||||
MergeTopkArrayTest(TOP_K, 0, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K, 0, NQ, TOP_K, false);
|
||||
MergeTopkArrayTest(0, TOP_K, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(0, TOP_K, NQ, TOP_K, false);
|
||||
|
||||
/* test2, id1/dist1 valid, id2/dist2 valid */
|
||||
MergeTopkArrayTest(TOP_K, TOP_K, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K, TOP_K, NQ, TOP_K, false);
|
||||
|
||||
/* test3, id1/dist1 small topk */
|
||||
MergeTopkArrayTest(TOP_K/2, TOP_K, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K/2, TOP_K, NQ, TOP_K, false);
|
||||
MergeTopkArrayTest(TOP_K, TOP_K/2, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K, TOP_K/2, NQ, TOP_K, false);
|
||||
|
||||
/* test4, id1/dist1 small topk, id2/dist2 small topk */
|
||||
MergeTopkArrayTest(TOP_K/2, TOP_K/3, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K/2, TOP_K/3, NQ, TOP_K, false);
|
||||
MergeTopkArrayTest(TOP_K/3, TOP_K/2, NQ, TOP_K, true);
|
||||
MergeTopkArrayTest(TOP_K/3, TOP_K/2, NQ, TOP_K, false);
|
||||
}
|
||||
// /* test2, id1/dist1 valid, id2/dist2 valid */
|
||||
// MergeTopkArrayTest(TOP_K, TOP_K, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K, TOP_K, NQ, TOP_K, false);
|
||||
//
|
||||
// /* test3, id1/dist1 small topk */
|
||||
// MergeTopkArrayTest(TOP_K/2, TOP_K, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K/2, TOP_K, NQ, TOP_K, false);
|
||||
// MergeTopkArrayTest(TOP_K, TOP_K/2, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K, TOP_K/2, NQ, TOP_K, false);
|
||||
//
|
||||
// /* test4, id1/dist1 small topk, id2/dist2 small topk */
|
||||
// MergeTopkArrayTest(TOP_K/2, TOP_K/3, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K/2, TOP_K/3, NQ, TOP_K, false);
|
||||
// MergeTopkArrayTest(TOP_K/3, TOP_K/2, NQ, TOP_K, true);
|
||||
// MergeTopkArrayTest(TOP_K/3, TOP_K/2, NQ, TOP_K, false);
|
||||
//}
|
||||
|
||||
TEST(DBSearchTest, REDUCE_PERF_TEST) {
|
||||
int32_t index_file_num = 478; /* sift1B dataset, index files num */
|
||||
|
|
@ -206,8 +225,8 @@ TEST(DBSearchTest, REDUCE_PERF_TEST) {
|
|||
std::vector<int32_t> thread_vec = {4, 8};
|
||||
std::vector<int32_t> nq_vec = {1, 10, 100};
|
||||
std::vector<int32_t> topk_vec = {1, 4, 16, 64};
|
||||
int32_t NQ = nq_vec[nq_vec.size()-1];
|
||||
int32_t TOPK = topk_vec[topk_vec.size()-1];
|
||||
int32_t NQ = nq_vec[nq_vec.size() - 1];
|
||||
int32_t TOPK = topk_vec[topk_vec.size() - 1];
|
||||
|
||||
std::vector<std::vector<int64_t>> id_vec;
|
||||
std::vector<std::vector<float>> dist_vec;
|
||||
|
|
@ -217,7 +236,7 @@ TEST(DBSearchTest, REDUCE_PERF_TEST) {
|
|||
|
||||
/* generate testing data */
|
||||
for (i = 0; i < index_file_num; i++) {
|
||||
BuildResult(input_ids, input_distance, TOPK, NQ, ascending);
|
||||
BuildResult(input_ids, input_distance, TOPK, TOPK, NQ, ascending);
|
||||
id_vec.push_back(input_ids);
|
||||
dist_vec.push_back(input_distance);
|
||||
}
|
||||
|
|
@ -237,7 +256,7 @@ TEST(DBSearchTest, REDUCE_PERF_TEST) {
|
|||
}
|
||||
|
||||
std::string str1 = "Method-1 " + std::to_string(max_thread_num) + " " +
|
||||
std::to_string(nq) + " " + std::to_string(top_k);
|
||||
std::to_string(nq) + " " + std::to_string(top_k);
|
||||
milvus::TimeRecorder rc1(str1);
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////
|
||||
|
|
@ -255,114 +274,114 @@ TEST(DBSearchTest, REDUCE_PERF_TEST) {
|
|||
|
||||
rc1.RecordSection("reduce done");
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////
|
||||
/* method-2 */
|
||||
std::vector<std::vector<int64_t>> id_vec_2(index_file_num);
|
||||
std::vector<std::vector<float>> dist_vec_2(index_file_num);
|
||||
std::vector<uint64_t> k_vec_2(index_file_num);
|
||||
for (i = 0; i < index_file_num; i++) {
|
||||
CopyResult(id_vec_2[i], dist_vec_2[i], top_k, id_vec[i], dist_vec[i], TOPK, nq);
|
||||
k_vec_2[i] = top_k;
|
||||
}
|
||||
|
||||
std::string str2 = "Method-2 " + std::to_string(max_thread_num) + " " +
|
||||
std::to_string(nq) + " " + std::to_string(top_k);
|
||||
milvus::TimeRecorder rc2(str2);
|
||||
|
||||
for (step = 1; step < index_file_num; step *= 2) {
|
||||
for (i = 0; i + step < index_file_num; i += step * 2) {
|
||||
ms::XSearchTask::MergeTopkArray(id_vec_2[i], dist_vec_2[i], k_vec_2[i],
|
||||
id_vec_2[i + step], dist_vec_2[i + step], k_vec_2[i + step],
|
||||
nq, top_k, ascending);
|
||||
}
|
||||
}
|
||||
ms::XSearchTask::MergeTopkToResultSet(id_vec_2[0],
|
||||
dist_vec_2[0],
|
||||
k_vec_2[0],
|
||||
nq,
|
||||
top_k,
|
||||
ascending,
|
||||
final_result_2);
|
||||
ASSERT_EQ(final_result_2.size(), nq);
|
||||
|
||||
rc2.RecordSection("reduce done");
|
||||
|
||||
for (i = 0; i < nq; i++) {
|
||||
ASSERT_EQ(final_result[i].size(), final_result_2[i].size());
|
||||
for (k = 0; k < final_result[i].size(); k++) {
|
||||
if (final_result[i][k].first != final_result_2[i][k].first) {
|
||||
std::cout << i << " " << k << std::endl;
|
||||
}
|
||||
ASSERT_EQ(final_result[i][k].first, final_result_2[i][k].first);
|
||||
ASSERT_EQ(final_result[i][k].second, final_result_2[i][k].second);
|
||||
}
|
||||
}
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////
|
||||
/* method-3 parallel */
|
||||
std::vector<std::vector<int64_t>> id_vec_3(index_file_num);
|
||||
std::vector<std::vector<float>> dist_vec_3(index_file_num);
|
||||
std::vector<uint64_t> k_vec_3(index_file_num);
|
||||
for (i = 0; i < index_file_num; i++) {
|
||||
CopyResult(id_vec_3[i], dist_vec_3[i], top_k, id_vec[i], dist_vec[i], TOPK, nq);
|
||||
k_vec_3[i] = top_k;
|
||||
}
|
||||
|
||||
std::string str3 = "Method-3 " + std::to_string(max_thread_num) + " " +
|
||||
std::to_string(nq) + " " + std::to_string(top_k);
|
||||
milvus::TimeRecorder rc3(str3);
|
||||
|
||||
for (step = 1; step < index_file_num; step *= 2) {
|
||||
for (i = 0; i + step < index_file_num; i += step * 2) {
|
||||
threads_list.push_back(
|
||||
threadPool.enqueue(ms::XSearchTask::MergeTopkArray,
|
||||
std::ref(id_vec_3[i]),
|
||||
std::ref(dist_vec_3[i]),
|
||||
std::ref(k_vec_3[i]),
|
||||
std::ref(id_vec_3[i + step]),
|
||||
std::ref(dist_vec_3[i + step]),
|
||||
std::ref(k_vec_3[i + step]),
|
||||
nq,
|
||||
top_k,
|
||||
ascending));
|
||||
}
|
||||
|
||||
while (threads_list.size() > 0) {
|
||||
int nready = 0;
|
||||
for (auto it = threads_list.begin(); it != threads_list.end(); it = it) {
|
||||
auto &p = *it;
|
||||
std::chrono::milliseconds span(0);
|
||||
if (p.wait_for(span) == std::future_status::ready) {
|
||||
threads_list.erase(it++);
|
||||
++nready;
|
||||
} else {
|
||||
++it;
|
||||
}
|
||||
}
|
||||
|
||||
if (nready == 0) {
|
||||
std::this_thread::yield();
|
||||
}
|
||||
}
|
||||
}
|
||||
ms::XSearchTask::MergeTopkToResultSet(id_vec_3[0],
|
||||
dist_vec_3[0],
|
||||
k_vec_3[0],
|
||||
nq,
|
||||
top_k,
|
||||
ascending,
|
||||
final_result_3);
|
||||
ASSERT_EQ(final_result_3.size(), nq);
|
||||
|
||||
rc3.RecordSection("reduce done");
|
||||
|
||||
for (i = 0; i < nq; i++) {
|
||||
ASSERT_EQ(final_result[i].size(), final_result_3[i].size());
|
||||
for (k = 0; k < final_result[i].size(); k++) {
|
||||
ASSERT_EQ(final_result[i][k].first, final_result_3[i][k].first);
|
||||
ASSERT_EQ(final_result[i][k].second, final_result_3[i][k].second);
|
||||
}
|
||||
}
|
||||
// ///////////////////////////////////////////////////////////////////////////////////////
|
||||
// /* method-2 */
|
||||
// std::vector<std::vector<int64_t>> id_vec_2(index_file_num);
|
||||
// std::vector<std::vector<float>> dist_vec_2(index_file_num);
|
||||
// std::vector<uint64_t> k_vec_2(index_file_num);
|
||||
// for (i = 0; i < index_file_num; i++) {
|
||||
// CopyResult(id_vec_2[i], dist_vec_2[i], top_k, id_vec[i], dist_vec[i], TOPK, nq);
|
||||
// k_vec_2[i] = top_k;
|
||||
// }
|
||||
//
|
||||
// std::string str2 = "Method-2 " + std::to_string(max_thread_num) + " " +
|
||||
// std::to_string(nq) + " " + std::to_string(top_k);
|
||||
// milvus::TimeRecorder rc2(str2);
|
||||
//
|
||||
// for (step = 1; step < index_file_num; step *= 2) {
|
||||
// for (i = 0; i + step < index_file_num; i += step * 2) {
|
||||
// ms::XSearchTask::MergeTopkArray(id_vec_2[i], dist_vec_2[i], k_vec_2[i],
|
||||
// id_vec_2[i + step], dist_vec_2[i + step], k_vec_2[i + step],
|
||||
// nq, top_k, ascending);
|
||||
// }
|
||||
// }
|
||||
// ms::XSearchTask::MergeTopkToResultSet(id_vec_2[0],
|
||||
// dist_vec_2[0],
|
||||
// k_vec_2[0],
|
||||
// nq,
|
||||
// top_k,
|
||||
// ascending,
|
||||
// final_result_2);
|
||||
// ASSERT_EQ(final_result_2.size(), nq);
|
||||
//
|
||||
// rc2.RecordSection("reduce done");
|
||||
//
|
||||
// for (i = 0; i < nq; i++) {
|
||||
// ASSERT_EQ(final_result[i].size(), final_result_2[i].size());
|
||||
// for (k = 0; k < final_result[i].size(); k++) {
|
||||
// if (final_result[i][k].first != final_result_2[i][k].first) {
|
||||
// std::cout << i << " " << k << std::endl;
|
||||
// }
|
||||
// ASSERT_EQ(final_result[i][k].first, final_result_2[i][k].first);
|
||||
// ASSERT_EQ(final_result[i][k].second, final_result_2[i][k].second);
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// ///////////////////////////////////////////////////////////////////////////////////////
|
||||
// /* method-3 parallel */
|
||||
// std::vector<std::vector<int64_t>> id_vec_3(index_file_num);
|
||||
// std::vector<std::vector<float>> dist_vec_3(index_file_num);
|
||||
// std::vector<uint64_t> k_vec_3(index_file_num);
|
||||
// for (i = 0; i < index_file_num; i++) {
|
||||
// CopyResult(id_vec_3[i], dist_vec_3[i], top_k, id_vec[i], dist_vec[i], TOPK, nq);
|
||||
// k_vec_3[i] = top_k;
|
||||
// }
|
||||
//
|
||||
// std::string str3 = "Method-3 " + std::to_string(max_thread_num) + " " +
|
||||
// std::to_string(nq) + " " + std::to_string(top_k);
|
||||
// milvus::TimeRecorder rc3(str3);
|
||||
//
|
||||
// for (step = 1; step < index_file_num; step *= 2) {
|
||||
// for (i = 0; i + step < index_file_num; i += step * 2) {
|
||||
// threads_list.push_back(
|
||||
// threadPool.enqueue(ms::XSearchTask::MergeTopkArray,
|
||||
// std::ref(id_vec_3[i]),
|
||||
// std::ref(dist_vec_3[i]),
|
||||
// std::ref(k_vec_3[i]),
|
||||
// std::ref(id_vec_3[i + step]),
|
||||
// std::ref(dist_vec_3[i + step]),
|
||||
// std::ref(k_vec_3[i + step]),
|
||||
// nq,
|
||||
// top_k,
|
||||
// ascending));
|
||||
// }
|
||||
//
|
||||
// while (threads_list.size() > 0) {
|
||||
// int nready = 0;
|
||||
// for (auto it = threads_list.begin(); it != threads_list.end(); it = it) {
|
||||
// auto &p = *it;
|
||||
// std::chrono::milliseconds span(0);
|
||||
// if (p.wait_for(span) == std::future_status::ready) {
|
||||
// threads_list.erase(it++);
|
||||
// ++nready;
|
||||
// } else {
|
||||
// ++it;
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// if (nready == 0) {
|
||||
// std::this_thread::yield();
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// ms::XSearchTask::MergeTopkToResultSet(id_vec_3[0],
|
||||
// dist_vec_3[0],
|
||||
// k_vec_3[0],
|
||||
// nq,
|
||||
// top_k,
|
||||
// ascending,
|
||||
// final_result_3);
|
||||
// ASSERT_EQ(final_result_3.size(), nq);
|
||||
//
|
||||
// rc3.RecordSection("reduce done");
|
||||
//
|
||||
// for (i = 0; i < nq; i++) {
|
||||
// ASSERT_EQ(final_result[i].size(), final_result_3[i].size());
|
||||
// for (k = 0; k < final_result[i].size(); k++) {
|
||||
// ASSERT_EQ(final_result[i][k].first, final_result_3[i][k].first);
|
||||
// ASSERT_EQ(final_result[i][k].second, final_result_3[i][k].second);
|
||||
// }
|
||||
// }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
numpy==1.16.3
|
||||
pymilvus>=0.1.18
|
||||
pyyaml==3.12
|
||||
pyyaml==5.1
|
||||
docker==4.0.2
|
||||
tableprint==0.8.0
|
||||
ansicolors==1.1.8
|
||||
|
|
@ -50,8 +50,7 @@ class TestAddBase:
|
|||
'''
|
||||
vector = gen_single_vector(dim)
|
||||
status, ids = connect.add_vectors(table, vector)
|
||||
ret = connect.has_table(table)
|
||||
assert ret == True
|
||||
assert assert_has_table(connect, table)
|
||||
|
||||
@pytest.mark.timeout(ADD_TIMEOUT)
|
||||
def test_delete_table_add_vector(self, connect, table):
|
||||
|
|
@ -618,8 +617,7 @@ class TestAddIP:
|
|||
'''
|
||||
vector = gen_single_vector(dim)
|
||||
status, ids = connect.add_vectors(ip_table, vector)
|
||||
ret = connect.has_table(ip_table)
|
||||
assert ret == True
|
||||
assert assert_has_table(connect, ip_table)
|
||||
|
||||
@pytest.mark.timeout(ADD_TIMEOUT)
|
||||
def test_delete_table_add_vector(self, connect, ip_table):
|
||||
|
|
|
|||
|
|
@ -1,425 +1,425 @@
|
|||
import time
|
||||
import random
|
||||
import pdb
|
||||
import logging
|
||||
import threading
|
||||
from builtins import Exception
|
||||
from multiprocessing import Pool, Process
|
||||
import pytest
|
||||
# import time
|
||||
# import random
|
||||
# import pdb
|
||||
# import logging
|
||||
# import threading
|
||||
# from builtins import Exception
|
||||
# from multiprocessing import Pool, Process
|
||||
# import pytest
|
||||
|
||||
from milvus import Milvus, IndexType
|
||||
from utils import *
|
||||
# from milvus import Milvus, IndexType
|
||||
# from utils import *
|
||||
|
||||
|
||||
dim = 128
|
||||
index_file_size = 10
|
||||
table_id = "test_delete"
|
||||
DELETE_TIMEOUT = 60
|
||||
vectors = gen_vectors(100, dim)
|
||||
# dim = 128
|
||||
# index_file_size = 10
|
||||
# table_id = "test_delete"
|
||||
# DELETE_TIMEOUT = 60
|
||||
# vectors = gen_vectors(100, dim)
|
||||
|
||||
class TestDeleteVectorsBase:
|
||||
"""
|
||||
generate invalid query range params
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_current_day(), get_current_day()),
|
||||
(get_last_day(1), get_last_day(1)),
|
||||
(get_next_day(1), get_next_day(1))
|
||||
]
|
||||
)
|
||||
def get_invalid_range(self, request):
|
||||
yield request.param
|
||||
# class TestDeleteVectorsBase:
|
||||
# """
|
||||
# generate invalid query range params
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_current_day(), get_current_day()),
|
||||
# (get_last_day(1), get_last_day(1)),
|
||||
# (get_next_day(1), get_next_day(1))
|
||||
# ]
|
||||
# )
|
||||
# def get_invalid_range(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_invalid_range(self, connect, table, get_invalid_range):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with invalid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_invalid_range[0]
|
||||
end_date = get_invalid_range[1]
|
||||
status, ids = connect.add_vectors(table, vectors)
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
assert not status.OK()
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_invalid_range(self, connect, table, get_invalid_range):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with invalid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_invalid_range[0]
|
||||
# end_date = get_invalid_range[1]
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# assert not status.OK()
|
||||
|
||||
"""
|
||||
generate valid query range params, no search result
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_last_day(2), get_last_day(1)),
|
||||
(get_last_day(2), get_current_day()),
|
||||
(get_next_day(1), get_next_day(2))
|
||||
]
|
||||
)
|
||||
def get_valid_range_no_result(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# generate valid query range params, no search result
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_last_day(2), get_last_day(1)),
|
||||
# (get_last_day(2), get_current_day()),
|
||||
# (get_next_day(1), get_next_day(2))
|
||||
# ]
|
||||
# )
|
||||
# def get_valid_range_no_result(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_no_result(self, connect, table, get_valid_range_no_result):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_valid_range_no_result[0]
|
||||
end_date = get_valid_range_no_result[1]
|
||||
status, ids = connect.add_vectors(table, vectors)
|
||||
time.sleep(2)
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(table)
|
||||
assert result == 100
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_no_result(self, connect, table, get_valid_range_no_result):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_valid_range_no_result[0]
|
||||
# end_date = get_valid_range_no_result[1]
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
# time.sleep(2)
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(table)
|
||||
# assert result == 100
|
||||
|
||||
"""
|
||||
generate valid query range params, no search result
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_last_day(2), get_next_day(2)),
|
||||
(get_current_day(), get_next_day(2)),
|
||||
]
|
||||
)
|
||||
def get_valid_range(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# generate valid query range params, no search result
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_last_day(2), get_next_day(2)),
|
||||
# (get_current_day(), get_next_day(2)),
|
||||
# ]
|
||||
# )
|
||||
# def get_valid_range(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range(self, connect, table, get_valid_range):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_valid_range[0]
|
||||
end_date = get_valid_range[1]
|
||||
status, ids = connect.add_vectors(table, vectors)
|
||||
time.sleep(2)
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(table)
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range(self, connect, table, get_valid_range):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_valid_range[0]
|
||||
# end_date = get_valid_range[1]
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
# time.sleep(2)
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(table)
|
||||
# assert result == 0
|
||||
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=gen_index_params()
|
||||
)
|
||||
def get_index_params(self, request, args):
|
||||
if "internal" not in args:
|
||||
if request.param["index_type"] == IndexType.IVF_SQ8H:
|
||||
pytest.skip("sq8h not support in open source")
|
||||
return request.param
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=gen_index_params()
|
||||
# )
|
||||
# def get_index_params(self, request, args):
|
||||
# if "internal" not in args:
|
||||
# if request.param["index_type"] == IndexType.IVF_SQ8H:
|
||||
# pytest.skip("sq8h not support in open source")
|
||||
# return request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_index_created(self, connect, table, get_index_params):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
index_params = get_index_params
|
||||
logging.getLogger().info(index_params)
|
||||
status, ids = connect.add_vectors(table, vectors)
|
||||
status = connect.create_index(table, index_params)
|
||||
logging.getLogger().info(status)
|
||||
logging.getLogger().info("Start delete vectors by range: %s:%s" % (start_date, end_date))
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(table)
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_index_created(self, connect, table, get_index_params):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# index_params = get_index_params
|
||||
# logging.getLogger().info(index_params)
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
# status = connect.create_index(table, index_params)
|
||||
# logging.getLogger().info(status)
|
||||
# logging.getLogger().info("Start delete vectors by range: %s:%s" % (start_date, end_date))
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(table)
|
||||
# assert result == 0
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_no_data(self, connect, table):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params, and no data in db
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
assert status.OK()
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_no_data(self, connect, table):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params, and no data in db
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# # status, ids = connect.add_vectors(table, vectors)
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_table_not_existed(self, connect):
|
||||
'''
|
||||
target: test delete vectors, table not existed in db
|
||||
method: call `delete_vectors_by_range`, with table not existed
|
||||
expected: return code not 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
table_name = gen_unique_str("not_existed_table")
|
||||
status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
assert not status.OK()
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_table_not_existed(self, connect):
|
||||
# '''
|
||||
# target: test delete vectors, table not existed in db
|
||||
# method: call `delete_vectors_by_range`, with table not existed
|
||||
# expected: return code not 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# table_name = gen_unique_str("not_existed_table")
|
||||
# status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
# assert not status.OK()
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_table_None(self, connect, table):
|
||||
'''
|
||||
target: test delete vectors, table set Nope
|
||||
method: call `delete_vectors_by_range`, with table value is None
|
||||
expected: return code not 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
table_name = None
|
||||
with pytest.raises(Exception) as e:
|
||||
status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_table_None(self, connect, table):
|
||||
# '''
|
||||
# target: test delete vectors, table set Nope
|
||||
# method: call `delete_vectors_by_range`, with table value is None
|
||||
# expected: return code not 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# table_name = None
|
||||
# with pytest.raises(Exception) as e:
|
||||
# status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_multi_tables(self, connect, get_valid_range):
|
||||
'''
|
||||
target: test delete vectors is correct or not with multiple tables of L2
|
||||
method: create 50 tables and add vectors into them , then delete vectors
|
||||
in valid range
|
||||
expected: return code 0
|
||||
'''
|
||||
nq = 100
|
||||
vectors = gen_vectors(nq, dim)
|
||||
table_list = []
|
||||
for i in range(50):
|
||||
table_name = gen_unique_str('test_delete_vectors_valid_range_multi_tables')
|
||||
table_list.append(table_name)
|
||||
param = {'table_name': table_name,
|
||||
'dimension': dim,
|
||||
'index_file_size': index_file_size,
|
||||
'metric_type': MetricType.L2}
|
||||
connect.create_table(param)
|
||||
status, ids = connect.add_vectors(table_name=table_name, records=vectors)
|
||||
time.sleep(2)
|
||||
start_date = get_valid_range[0]
|
||||
end_date = get_valid_range[1]
|
||||
for i in range(50):
|
||||
status = connect.delete_vectors_by_range(table_list[i], start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(table_list[i])
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_multi_tables(self, connect, get_valid_range):
|
||||
# '''
|
||||
# target: test delete vectors is correct or not with multiple tables of L2
|
||||
# method: create 50 tables and add vectors into them , then delete vectors
|
||||
# in valid range
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# nq = 100
|
||||
# vectors = gen_vectors(nq, dim)
|
||||
# table_list = []
|
||||
# for i in range(50):
|
||||
# table_name = gen_unique_str('test_delete_vectors_valid_range_multi_tables')
|
||||
# table_list.append(table_name)
|
||||
# param = {'table_name': table_name,
|
||||
# 'dimension': dim,
|
||||
# 'index_file_size': index_file_size,
|
||||
# 'metric_type': MetricType.L2}
|
||||
# connect.create_table(param)
|
||||
# status, ids = connect.add_vectors(table_name=table_name, records=vectors)
|
||||
# time.sleep(2)
|
||||
# start_date = get_valid_range[0]
|
||||
# end_date = get_valid_range[1]
|
||||
# for i in range(50):
|
||||
# status = connect.delete_vectors_by_range(table_list[i], start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(table_list[i])
|
||||
# assert result == 0
|
||||
|
||||
|
||||
class TestDeleteVectorsIP:
|
||||
"""
|
||||
generate invalid query range params
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_current_day(), get_current_day()),
|
||||
(get_last_day(1), get_last_day(1)),
|
||||
(get_next_day(1), get_next_day(1))
|
||||
]
|
||||
)
|
||||
def get_invalid_range(self, request):
|
||||
yield request.param
|
||||
# class TestDeleteVectorsIP:
|
||||
# """
|
||||
# generate invalid query range params
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_current_day(), get_current_day()),
|
||||
# (get_last_day(1), get_last_day(1)),
|
||||
# (get_next_day(1), get_next_day(1))
|
||||
# ]
|
||||
# )
|
||||
# def get_invalid_range(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_invalid_range(self, connect, ip_table, get_invalid_range):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with invalid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_invalid_range[0]
|
||||
end_date = get_invalid_range[1]
|
||||
status, ids = connect.add_vectors(ip_table, vectors)
|
||||
status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
assert not status.OK()
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_invalid_range(self, connect, ip_table, get_invalid_range):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with invalid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_invalid_range[0]
|
||||
# end_date = get_invalid_range[1]
|
||||
# status, ids = connect.add_vectors(ip_table, vectors)
|
||||
# status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
# assert not status.OK()
|
||||
|
||||
"""
|
||||
generate valid query range params, no search result
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_last_day(2), get_last_day(1)),
|
||||
(get_last_day(2), get_current_day()),
|
||||
(get_next_day(1), get_next_day(2))
|
||||
]
|
||||
)
|
||||
def get_valid_range_no_result(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# generate valid query range params, no search result
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_last_day(2), get_last_day(1)),
|
||||
# (get_last_day(2), get_current_day()),
|
||||
# (get_next_day(1), get_next_day(2))
|
||||
# ]
|
||||
# )
|
||||
# def get_valid_range_no_result(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_no_result(self, connect, ip_table, get_valid_range_no_result):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_valid_range_no_result[0]
|
||||
end_date = get_valid_range_no_result[1]
|
||||
status, ids = connect.add_vectors(ip_table, vectors)
|
||||
time.sleep(2)
|
||||
status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(ip_table)
|
||||
assert result == 100
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_no_result(self, connect, ip_table, get_valid_range_no_result):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_valid_range_no_result[0]
|
||||
# end_date = get_valid_range_no_result[1]
|
||||
# status, ids = connect.add_vectors(ip_table, vectors)
|
||||
# time.sleep(2)
|
||||
# status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(ip_table)
|
||||
# assert result == 100
|
||||
|
||||
"""
|
||||
generate valid query range params, no search result
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=[
|
||||
(get_last_day(2), get_next_day(2)),
|
||||
(get_current_day(), get_next_day(2)),
|
||||
]
|
||||
)
|
||||
def get_valid_range(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# generate valid query range params, no search result
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=[
|
||||
# (get_last_day(2), get_next_day(2)),
|
||||
# (get_current_day(), get_next_day(2)),
|
||||
# ]
|
||||
# )
|
||||
# def get_valid_range(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range(self, connect, ip_table, get_valid_range):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_valid_range[0]
|
||||
end_date = get_valid_range[1]
|
||||
status, ids = connect.add_vectors(ip_table, vectors)
|
||||
time.sleep(2)
|
||||
status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(ip_table)
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range(self, connect, ip_table, get_valid_range):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_valid_range[0]
|
||||
# end_date = get_valid_range[1]
|
||||
# status, ids = connect.add_vectors(ip_table, vectors)
|
||||
# time.sleep(2)
|
||||
# status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(ip_table)
|
||||
# assert result == 0
|
||||
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=gen_index_params()
|
||||
)
|
||||
def get_index_params(self, request, args):
|
||||
if "internal" not in args:
|
||||
if request.param["index_type"] == IndexType.IVF_SQ8H:
|
||||
pytest.skip("sq8h not support in open source")
|
||||
return request.param
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=gen_index_params()
|
||||
# )
|
||||
# def get_index_params(self, request, args):
|
||||
# if "internal" not in args:
|
||||
# if request.param["index_type"] == IndexType.IVF_SQ8H:
|
||||
# pytest.skip("sq8h not support in open source")
|
||||
# return request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_index_created(self, connect, ip_table, get_index_params):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
index_params = get_index_params
|
||||
logging.getLogger().info(index_params)
|
||||
status, ids = connect.add_vectors(ip_table, vectors)
|
||||
status = connect.create_index(ip_table, index_params)
|
||||
logging.getLogger().info(status)
|
||||
logging.getLogger().info("Start delete vectors by range: %s:%s" % (start_date, end_date))
|
||||
status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(ip_table)
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_index_created(self, connect, ip_table, get_index_params):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# index_params = get_index_params
|
||||
# logging.getLogger().info(index_params)
|
||||
# status, ids = connect.add_vectors(ip_table, vectors)
|
||||
# status = connect.create_index(ip_table, index_params)
|
||||
# logging.getLogger().info(status)
|
||||
# logging.getLogger().info("Start delete vectors by range: %s:%s" % (start_date, end_date))
|
||||
# status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(ip_table)
|
||||
# assert result == 0
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_no_data(self, connect, ip_table):
|
||||
'''
|
||||
target: test delete vectors, no index created
|
||||
method: call `delete_vectors_by_range`, with valid date params, and no data in db
|
||||
expected: return code 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
assert status.OK()
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_no_data(self, connect, ip_table):
|
||||
# '''
|
||||
# target: test delete vectors, no index created
|
||||
# method: call `delete_vectors_by_range`, with valid date params, and no data in db
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# # status, ids = connect.add_vectors(table, vectors)
|
||||
# status = connect.delete_vectors_by_range(ip_table, start_date, end_date)
|
||||
# assert status.OK()
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_table_None(self, connect, ip_table):
|
||||
'''
|
||||
target: test delete vectors, table set Nope
|
||||
method: call `delete_vectors_by_range`, with table value is None
|
||||
expected: return code not 0
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
table_name = None
|
||||
with pytest.raises(Exception) as e:
|
||||
status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_table_None(self, connect, ip_table):
|
||||
# '''
|
||||
# target: test delete vectors, table set Nope
|
||||
# method: call `delete_vectors_by_range`, with table value is None
|
||||
# expected: return code not 0
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# table_name = None
|
||||
# with pytest.raises(Exception) as e:
|
||||
# status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_valid_range_multi_tables(self, connect, get_valid_range):
|
||||
'''
|
||||
target: test delete vectors is correct or not with multiple tables of IP
|
||||
method: create 50 tables and add vectors into them , then delete vectors
|
||||
in valid range
|
||||
expected: return code 0
|
||||
'''
|
||||
nq = 100
|
||||
vectors = gen_vectors(nq, dim)
|
||||
table_list = []
|
||||
for i in range(50):
|
||||
table_name = gen_unique_str('test_delete_vectors_valid_range_multi_tables')
|
||||
table_list.append(table_name)
|
||||
param = {'table_name': table_name,
|
||||
'dimension': dim,
|
||||
'index_file_size': index_file_size,
|
||||
'metric_type': MetricType.IP}
|
||||
connect.create_table(param)
|
||||
status, ids = connect.add_vectors(table_name=table_name, records=vectors)
|
||||
time.sleep(2)
|
||||
start_date = get_valid_range[0]
|
||||
end_date = get_valid_range[1]
|
||||
for i in range(50):
|
||||
status = connect.delete_vectors_by_range(table_list[i], start_date, end_date)
|
||||
assert status.OK()
|
||||
status, result = connect.get_table_row_count(table_list[i])
|
||||
assert result == 0
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_valid_range_multi_tables(self, connect, get_valid_range):
|
||||
# '''
|
||||
# target: test delete vectors is correct or not with multiple tables of IP
|
||||
# method: create 50 tables and add vectors into them , then delete vectors
|
||||
# in valid range
|
||||
# expected: return code 0
|
||||
# '''
|
||||
# nq = 100
|
||||
# vectors = gen_vectors(nq, dim)
|
||||
# table_list = []
|
||||
# for i in range(50):
|
||||
# table_name = gen_unique_str('test_delete_vectors_valid_range_multi_tables')
|
||||
# table_list.append(table_name)
|
||||
# param = {'table_name': table_name,
|
||||
# 'dimension': dim,
|
||||
# 'index_file_size': index_file_size,
|
||||
# 'metric_type': MetricType.IP}
|
||||
# connect.create_table(param)
|
||||
# status, ids = connect.add_vectors(table_name=table_name, records=vectors)
|
||||
# time.sleep(2)
|
||||
# start_date = get_valid_range[0]
|
||||
# end_date = get_valid_range[1]
|
||||
# for i in range(50):
|
||||
# status = connect.delete_vectors_by_range(table_list[i], start_date, end_date)
|
||||
# assert status.OK()
|
||||
# status, result = connect.get_table_row_count(table_list[i])
|
||||
# assert result == 0
|
||||
|
||||
class TestDeleteVectorsParamsInvalid:
|
||||
# class TestDeleteVectorsParamsInvalid:
|
||||
|
||||
"""
|
||||
Test search table with invalid table names
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=gen_invalid_table_names()
|
||||
)
|
||||
def get_table_name(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# Test search table with invalid table names
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=gen_invalid_table_names()
|
||||
# )
|
||||
# def get_table_name(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.level(2)
|
||||
def test_delete_vectors_table_invalid_name(self, connect, get_table_name):
|
||||
'''
|
||||
'''
|
||||
start_date = get_current_day()
|
||||
end_date = get_next_day(2)
|
||||
table_name = get_table_name
|
||||
logging.getLogger().info(table_name)
|
||||
top_k = 1
|
||||
nprobe = 1
|
||||
status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
assert not status.OK()
|
||||
# @pytest.mark.level(2)
|
||||
# def test_delete_vectors_table_invalid_name(self, connect, get_table_name):
|
||||
# '''
|
||||
# '''
|
||||
# start_date = get_current_day()
|
||||
# end_date = get_next_day(2)
|
||||
# table_name = get_table_name
|
||||
# logging.getLogger().info(table_name)
|
||||
# top_k = 1
|
||||
# nprobe = 1
|
||||
# status = connect.delete_vectors_by_range(table_name, start_date, end_date)
|
||||
# assert not status.OK()
|
||||
|
||||
"""
|
||||
Test search table with invalid query ranges
|
||||
"""
|
||||
@pytest.fixture(
|
||||
scope="function",
|
||||
params=gen_invalid_query_ranges()
|
||||
)
|
||||
def get_query_ranges(self, request):
|
||||
yield request.param
|
||||
# """
|
||||
# Test search table with invalid query ranges
|
||||
# """
|
||||
# @pytest.fixture(
|
||||
# scope="function",
|
||||
# params=gen_invalid_query_ranges()
|
||||
# )
|
||||
# def get_query_ranges(self, request):
|
||||
# yield request.param
|
||||
|
||||
@pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
def test_delete_vectors_range_invalid(self, connect, table, get_query_ranges):
|
||||
'''
|
||||
target: test search fuction, with the wrong query_range
|
||||
method: search with query_range
|
||||
expected: raise an error, and the connection is normal
|
||||
'''
|
||||
start_date = get_query_ranges[0][0]
|
||||
end_date = get_query_ranges[0][1]
|
||||
status, ids = connect.add_vectors(table, vectors)
|
||||
logging.getLogger().info(get_query_ranges)
|
||||
with pytest.raises(Exception) as e:
|
||||
status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
# @pytest.mark.timeout(DELETE_TIMEOUT)
|
||||
# def test_delete_vectors_range_invalid(self, connect, table, get_query_ranges):
|
||||
# '''
|
||||
# target: test search fuction, with the wrong query_range
|
||||
# method: search with query_range
|
||||
# expected: raise an error, and the connection is normal
|
||||
# '''
|
||||
# start_date = get_query_ranges[0][0]
|
||||
# end_date = get_query_ranges[0][1]
|
||||
# status, ids = connect.add_vectors(table, vectors)
|
||||
# logging.getLogger().info(get_query_ranges)
|
||||
# with pytest.raises(Exception) as e:
|
||||
# status = connect.delete_vectors_by_range(table, start_date, end_date)
|
||||
|
|
@ -264,7 +264,7 @@ class TestTable:
|
|||
expected: status ok, and no table in tables
|
||||
'''
|
||||
status = connect.delete_table(table)
|
||||
assert not connect.has_table(table)
|
||||
assert not assert_has_table(connect, table)
|
||||
|
||||
def test_delete_table_ip(self, connect, ip_table):
|
||||
'''
|
||||
|
|
@ -274,7 +274,7 @@ class TestTable:
|
|||
expected: status ok, and no table in tables
|
||||
'''
|
||||
status = connect.delete_table(ip_table)
|
||||
assert not connect.has_table(ip_table)
|
||||
assert not assert_has_table(connect, ip_table)
|
||||
|
||||
@pytest.mark.level(2)
|
||||
def test_table_delete_without_connection(self, table, dis_connect):
|
||||
|
|
@ -314,7 +314,7 @@ class TestTable:
|
|||
connect.create_table(param)
|
||||
status = connect.delete_table(table_name)
|
||||
time.sleep(1)
|
||||
assert not connect.has_table(table_name)
|
||||
assert not assert_has_table(connect, table_name)
|
||||
|
||||
def test_delete_create_table_repeatedly(self, connect):
|
||||
'''
|
||||
|
|
@ -371,7 +371,7 @@ class TestTable:
|
|||
def deletetable(milvus):
|
||||
status = milvus.delete_table(table)
|
||||
# assert not status.code==0
|
||||
assert milvus.has_table(table)
|
||||
assert assert_has_table(milvus, table)
|
||||
assert status.OK()
|
||||
|
||||
for i in range(process_num):
|
||||
|
|
@ -411,11 +411,10 @@ class TestTable:
|
|||
def delete(connect,ids):
|
||||
i = 0
|
||||
while i < loop_num:
|
||||
# assert connect.has_table(table[ids*8+i])
|
||||
status = connect.delete_table(table[ids*process_num+i])
|
||||
time.sleep(2)
|
||||
assert status.OK()
|
||||
assert not connect.has_table(table[ids*process_num+i])
|
||||
assert not assert_has_table(connect, table[ids*process_num+i])
|
||||
i = i + 1
|
||||
|
||||
for i in range(process_num):
|
||||
|
|
@ -444,7 +443,7 @@ class TestTable:
|
|||
'index_file_size': index_file_size,
|
||||
'metric_type': MetricType.L2}
|
||||
connect.create_table(param)
|
||||
assert connect.has_table(table_name)
|
||||
assert assert_has_table(connect, table_name)
|
||||
|
||||
def test_has_table_ip(self, connect):
|
||||
'''
|
||||
|
|
@ -458,7 +457,7 @@ class TestTable:
|
|||
'index_file_size': index_file_size,
|
||||
'metric_type': MetricType.IP}
|
||||
connect.create_table(param)
|
||||
assert connect.has_table(table_name)
|
||||
assert assert_has_table(connect, table_name)
|
||||
|
||||
@pytest.mark.level(2)
|
||||
def test_has_table_without_connection(self, table, dis_connect):
|
||||
|
|
@ -468,7 +467,7 @@ class TestTable:
|
|||
expected: has table raise exception
|
||||
'''
|
||||
with pytest.raises(Exception) as e:
|
||||
status = dis_connect.has_table(table)
|
||||
assert_has_table(dis_connect, table)
|
||||
|
||||
def test_has_table_not_existed(self, connect):
|
||||
'''
|
||||
|
|
@ -478,7 +477,7 @@ class TestTable:
|
|||
expected: False
|
||||
'''
|
||||
table_name = gen_unique_str("test_table")
|
||||
assert not connect.has_table(table_name)
|
||||
assert not assert_has_table(connect, table_name)
|
||||
|
||||
"""
|
||||
******************************************************************
|
||||
|
|
@ -700,7 +699,7 @@ class TestCreateTableDimInvalid(object):
|
|||
'dimension': dimension,
|
||||
'index_file_size': index_file_size,
|
||||
'metric_type': MetricType.L2}
|
||||
if isinstance(dimension, int) and dimension > 0:
|
||||
if isinstance(dimension, int):
|
||||
status = connect.create_table(param)
|
||||
assert not status.OK()
|
||||
else:
|
||||
|
|
@ -778,7 +777,7 @@ def preload_table(connect, **params):
|
|||
return status
|
||||
|
||||
def has(connect, **params):
|
||||
status = connect.has_table(params["table_name"])
|
||||
status = assert_has_table(connect, params["table_name"])
|
||||
return status
|
||||
|
||||
def show(connect, **params):
|
||||
|
|
|
|||
|
|
@ -462,6 +462,11 @@ def gen_simple_index_params():
|
|||
return gen_params(index_types, nlists)
|
||||
|
||||
|
||||
def assert_has_table(conn, table_name):
|
||||
status, ok = conn.has_table(table_name)
|
||||
return status.OK() and ok
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import numpy
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue