2021-10-04 00:24:42 +00:00
# Time Travel Implementation (Segment Level)
2021-12-27 02:50:11 +00:00
Currently, there are two paths to implement time travel:
2021-10-04 00:24:42 +00:00
1. Restrict with vec_count, used in growing segment
2. Generate bitmask and combine it with DSL calculation results. It is mainly used in sealed segment
## Growing Segment Time Travel
1. When inserting, ensure that the inserted data is in ascending time order
2. Find the location of timestamp with binary search and record it as vec_count
2021-11-04 07:28:36 +00:00
3. Call vector_search interface and rest is handled inside segment, no need to handle bitmask generated by DSL
2021-10-04 00:24:42 +00:00
## SealedSegment Time Travel
2021-12-30 04:20:06 +00:00
1. During load, data is placed in a continuous memory area named chunk, with the following properties:
2021-10-04 00:24:42 +00:00
1. Data is divided into multiple segments
2021-11-17 11:29:19 +00:00
2. Data in one segment is ordered by primary key
2021-12-23 02:45:36 +00:00
3. Data between Segments is in timestamp order. That is, the timestamp of every entity in the previous segment must be less than the timestamp of the first entity in the next segment
2021-10-04 00:24:42 +00:00
2021-12-24 04:09:36 +00:00
2. The Algorithm for time travel is:
2021-10-04 00:24:42 +00:00
1. Use get_active_count interface, find the last segment containing a legal ts, and return the last element position of this segment as vec_count
2022-01-04 06:00:18 +00:00
2. Calculate the bitset mask with a timestamp. Due to the above properties, all the entities of the previous segment meet the conditions, and all the subsequent segments do not meet the conditions. Only the "last segment" needs to be calculated.
2021-11-04 07:28:36 +00:00
3. the calculated results of Bitset and DSL are combined and sent to vector search interface