ARMmbed
/
mbed-os
mirror of https://github.com/ARMmbed/mbed-os.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Squashed 'littlefs/' changes from 454b588..2ab150c 

2ab150c Removed toolchain specific warnings
0825d34 Adopted alternative implementation for lfs_ctz_index
46e22b2 Adopted lfs_ctz_index implementation using popcount
4fdca15 Slight name change with ctz skip-list functions

git-subtree-dir: littlefs
git-subtree-split: 2ab150cc500d5b8233ec8ef6109efa363bf1d38c
pull/6179/head
Christopher Haster 2017-10-30 18:16:52 -05:00
parent 3c3b0329d4
commit 0171b57a04
3 changed files with 253 additions and 168 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96
DESIGN.md
View File

@ -290,31 +290,32 @@ The path to data block 0 is even more quick, requiring only two jumps:
We can find the runtime complexity by looking at the path to any block from
the block containing the most pointers. Every step along the path divides
the search space for the block in half. This gives us a runtime of O(log n).
the search space for the block in half. This gives us a runtime of O(logn).
To get to the block with the most pointers, we can perform the same steps
backwards, which keeps the asymptotic runtime at O(log n). The interesting
backwards, which puts the runtime at O(2logn) = O(logn). The interesting
part about this data structure is that this optimal path occurs naturally
if we greedily choose the pointer that covers the most distance without passing
our target block.
So now we have a representation of files that can be appended trivially with
a runtime of O(1), and can be read with a worst case runtime of O(n logn).
a runtime of O(1), and can be read with a worst case runtime of O(nlogn).
Given that the the runtime is also divided by the amount of data we can store
in a block, this is pretty reasonable.
Unfortunately, the CTZ skip-list comes with a few questions that aren't
straightforward to answer. What is the overhead? How do we handle more
pointers than we can store in a block?
pointers than we can store in a block? How do we store the skip-list in
a directory entry?
One way to find the overhead per block is to look at the data structure as
multiple layers of linked-lists. Each linked-list skips twice as many blocks
as the previous linked-list. Or another way of looking at it is that each
as the previous linked-list. Another way of looking at it is that each
linked-list uses half as much storage per block as the previous linked-list.
As we approach infinity, the number of pointers per block forms a geometric
series. Solving this geometric series gives us an average of only 2 pointers
per block.
![overhead per block](https://latex.codecogs.com/gif.latex?%5Clim_%7Bn%5Cto%5Cinfty%7D%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D0%7D%5E%7Bn%7D%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%20%3D%20%5Csum_%7Bi%3D0%7D%5E%7B%5Cinfty%7D%5Cfrac%7B1%7D%7B2%5Ei%7D%20%3D%202)
![overhead_per_block](https://latex.codecogs.com/svg.latex?%5Clim_%7Bn%5Cto%5Cinfty%7D%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D0%7D%5E%7Bn%7D%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%20%3D%20%5Csum_%7Bi%3D0%7D%5Cfrac%7B1%7D%7B2%5Ei%7D%20%3D%202)
Finding the maximum number of pointers in a block is a bit more complicated,
but since our file size is limited by the integer width we use to store the
@ -322,7 +323,7 @@ size, we can solve for it. Setting the overhead of the maximum pointers equal
to the block size we get the following equation. Note that a smaller block size
results in more pointers, and a larger word width results in larger pointers.
![maximum overhead](https://latex.codecogs.com/gif.latex?B%20%3D%20%5Cfrac%7Bw%7D%7B8%7D%5Cleft%5Clceil%5Clog_2%5Cleft%28%5Cfrac%7B2%5Ew%7D%7BB-2%5Cfrac%7Bw%7D%7B8%7D%7D%5Cright%29%5Cright%5Crceil)
![maximum overhead](https://latex.codecogs.com/svg.latex?B%20%3D%20%5Cfrac%7Bw%7D%7B8%7D%5Cleft%5Clceil%5Clog_2%5Cleft%28%5Cfrac%7B2%5Ew%7D%7BB-2%5Cfrac%7Bw%7D%7B8%7D%7D%5Cright%29%5Cright%5Crceil)
where:
B = block size in bytes
@ -335,8 +336,83 @@ widths:
Since littlefs uses a 32 bit word size, we are limited to a minimum block
size of 104 bytes. This is a perfectly reasonable minimum block size, with most
block sizes starting around 512 bytes. So we can avoid the additional logic
needed to avoid overflowing our block's capacity in the CTZ skip-list.
block sizes starting around 512 bytes. So we can avoid additional logic to
avoid overflowing our block's capacity in the CTZ skip-list.
So, how do we store the skip-list in a directory entry? A naive approach would
be to store a pointer to the head of the skip-list, the length of the file
in bytes, the index of the head block in the skip-list, and the offset in the
head block in bytes. However this is a lot of information, and we can observe
that a file size maps to only one block index + offset pair. So it should be
sufficient to store only the pointer and file size.
But there is one problem, calculating the block index + offset pair from a
file size doesn't have an obvious implementation.
We can start by just writing down an equation. The first idea that comes to
mind is to just use a for loop to sum together blocks until we reach our
file size. We can write equation equation as a summation:
![summation1](https://latex.codecogs.com/svg.latex?N%20%3D%20%5Csum_i%5En%5Cleft%5BB-%5Cfrac%7Bw%7D%7B8%7D%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%5Cright%5D)
where:
B = block size in bytes
w = word width in bits
n = block index in skip-list
N = file size in bytes
And this works quite well, but is not trivial to calculate. This equation
requires O(n) to compute, which brings the entire runtime of reading a file
to O(n^2logn). Fortunately, the additional O(n) does not need to touch disk,
so it is not completely unreasonable. But if we could solve this equation into
a form that is easily computable, we can avoid a big slowdown.
Unfortunately, the summation of the CTZ instruction presents a big challenge.
How would you even begin to reason about integrating a bitwise instruction?
Fortunately, there is a powerful tool I've found useful in these situations:
The [On-Line Encyclopedia of Integer Sequences (OEIS)](https://oeis.org/).
If we work out the first couple of values in our summation, we find that CTZ
maps to [A001511](https://oeis.org/A001511), and its partial summation maps
to [A005187](https://oeis.org/A005187), and surprisingly, both of these
sequences have relatively trivial equations! This leads us to the completely
unintuitive property:
![mindblown](https://latex.codecogs.com/svg.latex?%5Csum_i%5En%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%20%3D%202n-%5Ctext%7Bpopcount%7D%28n%29)
where:
ctz(i) = the number of trailing bits that are 0 in i
popcount(i) = the number of bits that are 1 in i
I find it bewildering that these two seemingly unrelated bitwise instructions
are related by this property. But if we start to disect this equation we can
see that it does hold. As n approaches infinity, we do end up with an average
overhead of 2 pointers as we find earlier. And popcount seems to handle the
error from this average as it accumulates in the CTZ skip-list.
Now we can substitute into the original equation to get a trivial equation
for a file size:
![summation2](https://latex.codecogs.com/svg.latex?N%20%3D%20Bn%20-%20%5Cfrac%7Bw%7D%7B8%7D%5Cleft%282n-%5Ctext%7Bpopcount%7D%28n%29%5Cright%29)
Unfortunately, we're not quite done. The popcount function is non-injective,
so we can only find the file size from the block index, not the other way
around. However, we can solve for an n' block index that is greater than n
with an error bounded by the range of the popcount function. We can then
repeatedly substitute this n' into the original equation until the error
is smaller than the integer division. As it turns out, we only need to
perform this substitution once. Now we directly calculate our block index:
![formulaforn](https://latex.codecogs.com/svg.latex?n%20%3D%20%5Cleft%5Clfloor%5Cfrac%7BN-%5Cfrac%7Bw%7D%7B8%7D%5Cleft%28%5Ctext%7Bpopcount%7D%5Cleft%28%5Cfrac%7BN%7D%7BB-2%5Cfrac%7Bw%7D%7B8%7D%7D-1%5Cright%29+2%5Cright%29%7D%7BB-2%5Cfrac%7Bw%7D%7B8%7D%7D%5Cright%5Crfloor)
Now that we have our block index n, we can just plug it back into the above
equation to find the offset. However, we do need to rearrange the equation
a bit to avoid integer overflow:
![formulaforoff](https://latex.codecogs.com/svg.latex?%5Cmathit%7Boff%7D%20%3D%20N%20-%20%5Cleft%28B-2%5Cfrac%7Bw%7D%7B8%7D%5Cright%29n%20-%20%5Cfrac%7Bw%7D%7B8%7D%5Ctext%7Bpopcount%7D%28n%29)
The solution involves quite a bit of math, but computers are very good at math.
We can now solve for the block index + offset while only needed to store the
file size in O(1).
Here is what it might look like to update a file stored with a CTZ skip-list:
```
@ -1129,7 +1205,7 @@ So, to summarize:
metadata block is active
4. Directory blocks contain either references to other directories or files
5. Files are represented by copy-on-write CTZ skip-lists which support O(1)
append and O(n logn) reading
append and O(nlogn) reading
6. Blocks are allocated by scanning the filesystem for used blocks in a
fixed-size lookahead region is that stored in a bit-vector
7. To facilitate scanning the filesystem, all directories are part of a

77
lfs.c
View File

@ -373,8 +373,8 @@ static int lfs_dir_alloc(lfs_t *lfs, lfs_dir_t *dir) {
// set defaults
dir->d.rev += 1;
dir->d.size = sizeof(dir->d)+4;
dir->d.tail[0] = -1;
dir->d.tail[1] = -1;
dir->d.tail[0] = 0xffffffff;
dir->d.tail[1] = 0xffffffff;
dir->off = sizeof(dir->d);
// don't write out yet, let caller take care of that
@ -455,6 +455,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
bool relocated = false;
while (true) {
if (true) {
int err = lfs_bd_erase(lfs, dir->pair[0]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
@ -530,7 +531,8 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
// successful commit, check checksum to make sure
crc = 0xffffffff;
err = lfs_bd_crc(lfs, dir->pair[0], 0, 0x7fffffff & dir->d.size, &crc);
err = lfs_bd_crc(lfs, dir->pair[0], 0,
0x7fffffff & dir->d.size, &crc);
if (err) {
return err;
}
@ -538,6 +540,7 @@ static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
if (crc == 0) {
break;
}
}
relocate:
//commit was corrupted
@ -554,7 +557,7 @@ relocate:
}
// relocate half of pair
err = lfs_alloc(lfs, &dir->pair[0]);
int err = lfs_alloc(lfs, &dir->pair[0]);
if (err) {
return err;
}
@ -791,8 +794,6 @@ static int lfs_dir_find(lfs_t *lfs, lfs_dir_t *dir,
return err;
}
}
return 0;
}
@ -1003,30 +1004,31 @@ int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
/// File index list operations ///
static int lfs_index(lfs_t *lfs, lfs_off_t *off) {
lfs_off_t i = 0;
while (*off >= lfs->cfg->block_size) {
i += 1;
*off -= lfs->cfg->block_size;
*off += 4*(lfs_ctz(i) + 1);
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
lfs_off_t size = *off;
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
if (i == 0) {
return 0;
}
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
}
static int lfs_index_find(lfs_t *lfs,
static int lfs_ctz_find(lfs_t *lfs,
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
*block = -1;
*block = 0xffffffff;
*off = 0;
return 0;
}
lfs_off_t current = lfs_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_index(lfs, &pos);
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
while (current > target) {
lfs_size_t skip = lfs_min(
@ -1047,11 +1049,12 @@ static int lfs_index_find(lfs_t *lfs,
return 0;
}
static int lfs_index_extend(lfs_t *lfs,
static int lfs_ctz_extend(lfs_t *lfs,
lfs_cache_t *rcache, lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
lfs_off_t *block, lfs_block_t *off) {
while (true) {
if (true) {
// go ahead and grab a block
int err = lfs_alloc(lfs, block);
if (err) {
@ -1073,19 +1076,21 @@ static int lfs_index_extend(lfs_t *lfs,
}
size -= 1;
lfs_off_t index = lfs_index(lfs, &size);
lfs_off_t index = lfs_ctz_index(lfs, &size);
size += 1;
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
int err = lfs_cache_read(lfs, rcache, NULL, head, i, &data, 1);
int err = lfs_cache_read(lfs, rcache, NULL,
head, i, &data, 1);
if (err) {
return err;
}
err = lfs_cache_prog(lfs, pcache, rcache, *block, i, &data, 1);
err = lfs_cache_prog(lfs, pcache, rcache,
*block, i, &data, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
@ -1113,7 +1118,8 @@ static int lfs_index_extend(lfs_t *lfs,
}
if (i != skips-1) {
err = lfs_cache_read(lfs, rcache, NULL, head, 4*i, &head, 4);
err = lfs_cache_read(lfs, rcache, NULL,
head, 4*i, &head, 4);
if (err) {
return err;
}
@ -1124,6 +1130,7 @@ static int lfs_index_extend(lfs_t *lfs,
*off = 4*skips;
return 0;
}
relocate:
LFS_DEBUG("Bad block at %d", *block);
@ -1133,7 +1140,7 @@ relocate:
}
}
static int lfs_index_traverse(lfs_t *lfs,
static int lfs_ctz_traverse(lfs_t *lfs,
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
int (*cb)(void*, lfs_block_t), void *data) {
@ -1141,7 +1148,7 @@ static int lfs_index_traverse(lfs_t *lfs,
return 0;
}
lfs_off_t index = lfs_index(lfs, &(lfs_off_t){size-1});
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
while (true) {
int err = cb(data, head);
@ -1160,8 +1167,6 @@ static int lfs_index_traverse(lfs_t *lfs,
index -= 1;
}
return 0;
}
@ -1199,7 +1204,7 @@ int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
entry.d.elen = sizeof(entry.d) - 4;
entry.d.alen = 0;
entry.d.nlen = strlen(path);
entry.d.u.file.head = -1;
entry.d.u.file.head = 0xffffffff;
entry.d.u.file.size = 0;
err = lfs_dir_append(lfs, &cwd, &entry, path);
if (err) {
@ -1221,7 +1226,7 @@ int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
file->pos = 0;
if (flags & LFS_O_TRUNC) {
file->head = -1;
file->head = 0xffffffff;
file->size = 0;
}
@ -1455,7 +1460,7 @@ lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
// check if we need a new block
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
int err = lfs_index_find(lfs, &file->cache, NULL,
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos, &file->block, &file->off);
if (err) {
@ -1522,7 +1527,7 @@ lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
file->off == lfs->cfg->block_size) {
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
int err = lfs_index_find(lfs, &file->cache, NULL,
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos-1, &file->block, &file->off);
if (err) {
@ -1535,7 +1540,7 @@ lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
// extend file with new blocks
lfs_alloc_ack(lfs);
int err = lfs_index_extend(lfs, &lfs->rcache, &file->cache,
int err = lfs_ctz_extend(lfs, &lfs->rcache, &file->cache,
file->block, file->pos,
&file->block, &file->off);
if (err) {
@ -1588,13 +1593,13 @@ lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
if (whence == LFS_SEEK_SET) {
file->pos = off;
} else if (whence == LFS_SEEK_CUR) {
if (-off > file->pos) {
if ((lfs_off_t)-off > file->pos) {
return LFS_ERR_INVAL;
}
file->pos = file->pos + off;
} else if (whence == LFS_SEEK_END) {
if (-off > file->size) {
if ((lfs_off_t)-off > file->size) {
return LFS_ERR_INVAL;
}
@ -2070,7 +2075,7 @@ int lfs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
dir.off += lfs_entry_size(&entry);
if ((0x70 & entry.d.type) == (0x70 & LFS_TYPE_REG)) {
int err = lfs_index_traverse(lfs, &lfs->rcache, NULL,
int err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
entry.d.u.file.head, entry.d.u.file.size, cb, data);
if (err) {
return err;
@ -2089,7 +2094,7 @@ int lfs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if (f->flags & LFS_F_DIRTY) {
int err = lfs_index_traverse(lfs, &lfs->rcache, &f->cache,
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->head, f->size, cb, data);
if (err) {
return err;
@ -2097,7 +2102,7 @@ int lfs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
}
if (f->flags & LFS_F_WRITING) {
int err = lfs_index_traverse(lfs, &lfs->rcache, &f->cache,
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->block, f->pos, cb, data);
if (err) {
return err;

4
lfs_util.h
View File

@ -41,6 +41,10 @@ static inline uint32_t lfs_npw2(uint32_t a) {
return 32 - __builtin_clz(a-1);
}
static inline uint32_t lfs_popc(uint32_t a) {
return __builtin_popcount(a);
}
static inline int lfs_scmp(uint32_t a, uint32_t b) {
return (int)(unsigned)(a - b);
}