ARMmbed
/
mbed-os
mirror of https://github.com/ARMmbed/mbed-os.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge commit '0a197465d33900a5ac2ef41486410d44f3d15ad3'

pull/5538/head
Christopher Haster 2017-11-20 00:16:47 -06:00
parent 3c9f2be163 0a197465d3
commit 9b79136ea7
2 changed files with 92 additions and 76 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

93
littlefs/DESIGN.md
View File

@ -1,6 +1,6 @@
## The design of the little filesystem ## The design of the little filesystem
The littlefs is a little fail-safe filesystem designed for embedded systems. A little fail-safe filesystem designed for embedded systems.
``` ```
| | | .---._____ | | | .---._____
@ -16,9 +16,9 @@ more about filesystem design by tackling the relative unsolved problem of
managing a robust filesystem resilient to power loss on devices managing a robust filesystem resilient to power loss on devices
with limited RAM and ROM. with limited RAM and ROM.
The embedded systems the littlefs is targeting are usually 32bit The embedded systems the littlefs is targeting are usually 32 bit
microcontrollers with around 32Kbytes of RAM and 512Kbytes of ROM. These are microcontrollers with around 32KB of RAM and 512KB of ROM. These are
often paired with SPI NOR flash chips with about 4Mbytes of flash storage. often paired with SPI NOR flash chips with about 4MB of flash storage.
Flash itself is a very interesting piece of technology with quite a bit of Flash itself is a very interesting piece of technology with quite a bit of
nuance. Unlike most other forms of storage, writing to flash requires two nuance. Unlike most other forms of storage, writing to flash requires two
@ -32,17 +32,17 @@ has more information if you are interesting in how this works.
This leaves us with an interesting set of limitations that can be simplified This leaves us with an interesting set of limitations that can be simplified
to three strong requirements: to three strong requirements:
1. **Fail-safe** - This is actually the main goal of the littlefs and the focus 1. **Power-loss resilient** - This is the main goal of the littlefs and the
of this project. Embedded systems are usually designed without a shutdown focus of this project. Embedded systems are usually designed without a
routine and a notable lack of user interface for recovery, so filesystems shutdown routine and a notable lack of user interface for recovery, so
targeting embedded systems should be prepared to lose power an any given filesystems targeting embedded systems must be prepared to lose power an
time. any given time.
Despite this state of things, there are very few embedded filesystems that Despite this state of things, there are very few embedded filesystems that
handle power loss in a reasonable manner, and can become corrupted if the handle power loss in a reasonable manner, and most can become corrupted if
user is unlucky enough. the user is unlucky enough.
2. **Wear awareness** - Due to the destructive nature of flash, most flash 2. **Wear leveling** - Due to the destructive nature of flash, most flash
chips have a limited number of erase cycles, usually in the order of around chips have a limited number of erase cycles, usually in the order of around
100,000 erases per block for NOR flash. Filesystems that don't take wear 100,000 erases per block for NOR flash. Filesystems that don't take wear
into account can easily burn through blocks used to store frequently updated into account can easily burn through blocks used to store frequently updated
@ -78,9 +78,9 @@ summary of the general ideas behind some of them.
Most of the existing filesystems fall into the one big category of filesystem Most of the existing filesystems fall into the one big category of filesystem
designed in the early days of spinny magnet disks. While there is a vast amount designed in the early days of spinny magnet disks. While there is a vast amount
of interesting technology and ideas in this area, the nature of spinny magnet of interesting technology and ideas in this area, the nature of spinny magnet
disks encourage properties such as grouping writes near each other, that don't disks encourage properties, such as grouping writes near each other, that don't
make as much sense on recent storage types. For instance, on flash, write make as much sense on recent storage types. For instance, on flash, write
locality is not as important and can actually increase wear destructively. locality is not important and can actually increase wear destructively.
One of the most popular designs for flash filesystems is called the One of the most popular designs for flash filesystems is called the
[logging filesystem](https://en.wikipedia.org/wiki/Log-structured_file_system). [logging filesystem](https://en.wikipedia.org/wiki/Log-structured_file_system).
@ -97,8 +97,7 @@ scaling as the size of storage increases. And most filesystems compensate by
caching large parts of the filesystem in RAM, a strategy that is unavailable caching large parts of the filesystem in RAM, a strategy that is unavailable
for embedded systems. for embedded systems.
Another interesting filesystem design technique that the littlefs borrows the Another interesting filesystem design technique is that of [copy-on-write (COW)](https://en.wikipedia.org/wiki/Copy-on-write).
most from, is the [copy-on-write (COW)](https://en.wikipedia.org/wiki/Copy-on-write).
A good example of this is the [btrfs](https://en.wikipedia.org/wiki/Btrfs) A good example of this is the [btrfs](https://en.wikipedia.org/wiki/Btrfs)
filesystem. COW filesystems can easily recover from corrupted blocks and have filesystem. COW filesystems can easily recover from corrupted blocks and have
natural protection against power loss. However, if they are not designed with natural protection against power loss. However, if they are not designed with
@ -150,12 +149,12 @@ check our checksum we notice that block 1 was corrupted. So we fall back to
block 2 and use the value 9. block 2 and use the value 9.
Using this concept, the littlefs is able to update metadata blocks atomically. Using this concept, the littlefs is able to update metadata blocks atomically.
There are a few other tweaks, such as using a 32bit crc and using sequence There are a few other tweaks, such as using a 32 bit crc and using sequence
arithmetic to handle revision count overflow, but the basic concept arithmetic to handle revision count overflow, but the basic concept
is the same. These metadata pairs define the backbone of the littlefs, and the is the same. These metadata pairs define the backbone of the littlefs, and the
rest of the filesystem is built on top of these atomic updates. rest of the filesystem is built on top of these atomic updates.
## Files ## Non-meta data
Now, the metadata pairs do come with some drawbacks. Most notably, each pair Now, the metadata pairs do come with some drawbacks. Most notably, each pair
requires two blocks for each block of data. I'm sure users would be very requires two blocks for each block of data. I'm sure users would be very
@ -224,12 +223,12 @@ Exhibit A: A linked-list
To get around this, the littlefs, at its heart, stores files backwards. Each To get around this, the littlefs, at its heart, stores files backwards. Each
block points to its predecessor, with the first block containing no pointers. block points to its predecessor, with the first block containing no pointers.
If you think about this, it makes a bit of sense. Appending blocks just point If you think about for a while, it starts to make a bit of sense. Appending
to their predecessor and no other blocks need to be updated. If we update blocks just point to their predecessor and no other blocks need to be updated.
a block in the middle, we will need to copy out the blocks that follow, If we update a block in the middle, we will need to copy out the blocks that
but can reuse the blocks before the modified block. Since most file operations follow, but can reuse the blocks before the modified block. Since most file
either reset the file each write or append to files, this design avoids operations either reset the file each write or append to files, this design
copying the file in the most common cases. avoids copying the file in the most common cases.
``` ```
Exhibit B: A backwards linked-list Exhibit B: A backwards linked-list
@ -351,7 +350,7 @@ file size doesn't have an obvious implementation.
We can start by just writing down an equation. The first idea that comes to We can start by just writing down an equation. The first idea that comes to
mind is to just use a for loop to sum together blocks until we reach our mind is to just use a for loop to sum together blocks until we reach our
file size. We can write equation equation as a summation: file size. We can write this equation as a summation:
![summation1](https://latex.codecogs.com/svg.latex?N%20%3D%20%5Csum_i%5En%5Cleft%5BB-%5Cfrac%7Bw%7D%7B8%7D%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%5Cright%5D) ![summation1](https://latex.codecogs.com/svg.latex?N%20%3D%20%5Csum_i%5En%5Cleft%5BB-%5Cfrac%7Bw%7D%7B8%7D%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%5Cright%5D)
@ -374,7 +373,7 @@ The [On-Line Encyclopedia of Integer Sequences (OEIS)](https://oeis.org/).
If we work out the first couple of values in our summation, we find that CTZ If we work out the first couple of values in our summation, we find that CTZ
maps to [A001511](https://oeis.org/A001511), and its partial summation maps maps to [A001511](https://oeis.org/A001511), and its partial summation maps
to [A005187](https://oeis.org/A005187), and surprisingly, both of these to [A005187](https://oeis.org/A005187), and surprisingly, both of these
sequences have relatively trivial equations! This leads us to the completely sequences have relatively trivial equations! This leads us to a rather
unintuitive property: unintuitive property:
![mindblown](https://latex.codecogs.com/svg.latex?%5Csum_i%5En%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%20%3D%202n-%5Ctext%7Bpopcount%7D%28n%29) ![mindblown](https://latex.codecogs.com/svg.latex?%5Csum_i%5En%5Cleft%28%5Ctext%7Bctz%7D%28i%29+1%5Cright%29%20%3D%202n-%5Ctext%7Bpopcount%7D%28n%29)
@ -383,7 +382,7 @@ where:
ctz(i) = the number of trailing bits that are 0 in i ctz(i) = the number of trailing bits that are 0 in i
popcount(i) = the number of bits that are 1 in i popcount(i) = the number of bits that are 1 in i
I find it bewildering that these two seemingly unrelated bitwise instructions It's a bit bewildering that these two seemingly unrelated bitwise instructions
are related by this property. But if we start to disect this equation we can are related by this property. But if we start to disect this equation we can
see that it does hold. As n approaches infinity, we do end up with an average see that it does hold. As n approaches infinity, we do end up with an average
overhead of 2 pointers as we find earlier. And popcount seems to handle the overhead of 2 pointers as we find earlier. And popcount seems to handle the
@ -1154,21 +1153,26 @@ develops errors and needs to be moved.
The second concern for the littlefs, is that blocks in the filesystem may wear The second concern for the littlefs, is that blocks in the filesystem may wear
unevenly. In this situation, a filesystem may meet an early demise where unevenly. In this situation, a filesystem may meet an early demise where
there are no more non-corrupted blocks that aren't in use. It may be entirely there are no more non-corrupted blocks that aren't in use. It's common to
possible that files were written once and left unmodified, wasting the have files that were written once and left unmodified, wasting the potential
potential erase cycles of the blocks it sits on. erase cycles of the blocks it sits on.
Wear leveling is a term that describes distributing block writes evenly to Wear leveling is a term that describes distributing block writes evenly to
avoid the early termination of a flash part. There are typically two levels avoid the early termination of a flash part. There are typically two levels
of wear leveling: of wear leveling:
1. Dynamic wear leveling - Blocks are distributed evenly during blocks writes. 1. Dynamic wear leveling - Wear is distributed evenly across all **dynamic**
Note that the issue with write-once files still exists in this case. blocks. Usually this is accomplished by simply choosing the unused block
2. Static wear leveling - Unmodified blocks are evicted for new block writes. with the lowest amount of wear. Note this does not solve the problem of
This provides the longest lifetime for a flash device. static data.
2. Static wear leveling - Wear is distributed evenly across all **dynamic**
and **static** blocks. Unmodified blocks may be evicted for new block
writes. This does handle the problem of static data but may lead to
wear amplification.
Now, it's possible to use the revision count on metadata pairs to approximate In littlefs's case, it's possible to use the revision count on metadata pairs
the wear of a metadata block. And combined with the COW nature of files, the to approximate the wear of a metadata block. And combined with the COW nature
littlefs could provide a form of dynamic wear leveling. of files, littlefs could provide your usually implementation of dynamic wear
leveling.
However, the littlefs does not. This is for a few reasons. Most notably, even However, the littlefs does not. This is for a few reasons. Most notably, even
if the littlefs did implement dynamic wear leveling, this would still not if the littlefs did implement dynamic wear leveling, this would still not
@ -1179,19 +1183,20 @@ As a flash device reaches the end of its life, the metadata blocks will
naturally be the first to go since they are updated most often. In this naturally be the first to go since they are updated most often. In this
situation, the littlefs is designed to simply move on to another set of situation, the littlefs is designed to simply move on to another set of
metadata blocks. This travelling means that at the end of a flash device's metadata blocks. This travelling means that at the end of a flash device's
life, the filesystem will have worn the device down as evenly as a dynamic life, the filesystem will have worn the device down nearly as evenly as the
wear leveling filesystem could anyways. Simply put, if the lifetime of flash usual dynamic wear leveling could. More aggressive wear leveling would come
is a serious concern, static wear leveling is the only valid solution. with a code-size cost for marginal benefit.
This is a very important takeaway to note. If your storage stack uses highly
sensitive storage such as NAND flash. In most cases you are going to be better One important takeaway to note, if your storage stack uses highly sensitive
off just using a [flash translation layer (FTL)](https://en.wikipedia.org/wiki/Flash_translation_layer). storage such as NAND flash, static wear leveling is the only valid solution.
In most cases you are going to be better off using a full [flash translation layer (FTL)](https://en.wikipedia.org/wiki/Flash_translation_layer).
NAND flash already has many limitations that make it poorly suited for an NAND flash already has many limitations that make it poorly suited for an
embedded system: low erase cycles, very large blocks, errors that can develop embedded system: low erase cycles, very large blocks, errors that can develop
even during reads, errors that can develop during writes of neighboring blocks. even during reads, errors that can develop during writes of neighboring blocks.
Managing sensitive storage such as NAND flash is out of scope for the littlefs. Managing sensitive storage such as NAND flash is out of scope for the littlefs.
The littlefs does have some properties that may be beneficial on top of a FTL, The littlefs does have some properties that may be beneficial on top of a FTL,
such as limiting the number of writes where possible. But if you have the such as limiting the number of writes where possible, but if you have the
storage requirements that necessitate the need of NAND flash, you should have storage requirements that necessitate the need of NAND flash, you should have
the RAM to match and just use an FTL or flash filesystem. the RAM to match and just use an FTL or flash filesystem.

75
littlefs/README.md
View File

@ -11,23 +11,17 @@ A little fail-safe filesystem designed for embedded systems.
| | | | | |
``` ```
**Fail-safe** - The littlefs is designed to work consistently with random **Bounded RAM/ROM** - The littlefs is designed to work with a limited amount
power failures. During filesystem operations the storage on disk is always of memory. Recursion is avoided and dynamic memory is limited to configurable
kept in a valid state. The filesystem also has strong copy-on-write garuntees. buffers that can be provided statically.
When updating a file, the original file will remain unmodified until the
file is closed, or sync is called.
**Wear awareness** - While the littlefs does not implement static wear **Power-loss resilient** - The littlefs is designed for systems that may have
leveling, the littlefs takes into account write errors reported by the random power failures. The littlefs has strong copy-on-write guaruntees and
underlying block device and uses a limited form of dynamic wear leveling storage on disk is always kept in a valid state.
to manage blocks that go bad during the lifetime of the filesystem.
**Bounded ram/rom** - The littlefs is designed to work in a **Wear leveling** - Since the most common form of embedded storage is erodible
limited amount of memory, recursion is avoided, and dynamic memory is kept flash memories, littlefs provides a form of dynamic wear leveling for systems
to a minimum. The littlefs allocates two fixed-size buffers for general that can not fit a full flash translation layer.
operations, and one fixed-size buffer per file. If there is only ever one file
in use, all memory can be provided statically and the littlefs can be used
in a system without dynamic memory.
## Example ## Example
@ -96,7 +90,7 @@ int main(void) {
Detailed documentation (or at least as much detail as is currently available) Detailed documentation (or at least as much detail as is currently available)
can be cound in the comments in [lfs.h](lfs.h). can be cound in the comments in [lfs.h](lfs.h).
As you may have noticed, the littlefs takes in a configuration structure that As you may have noticed, littlefs takes in a configuration structure that
defines how the filesystem operates. The configuration struct provides the defines how the filesystem operates. The configuration struct provides the
filesystem with the block device operations and dimensions, tweakable filesystem with the block device operations and dimensions, tweakable
parameters that tradeoff memory usage for performance, and optional parameters that tradeoff memory usage for performance, and optional
@ -104,14 +98,16 @@ static buffers if the user wants to avoid dynamic memory.
The state of the littlefs is stored in the `lfs_t` type which is left up The state of the littlefs is stored in the `lfs_t` type which is left up
to the user to allocate, allowing multiple filesystems to be in use to the user to allocate, allowing multiple filesystems to be in use
simultaneously. With the `lfs_t` and configuration struct, a user can either simultaneously. With the `lfs_t` and configuration struct, a user can
format a block device or mount the filesystem. format a block device or mount the filesystem.
Once mounted, the littlefs provides a full set of posix-like file and Once mounted, the littlefs provides a full set of posix-like file and
directory functions, with the deviation that the allocation of filesystem directory functions, with the deviation that the allocation of filesystem
structures must be provided by the user. An important addition is that structures must be provided by the user.
no file updates will actually be written to disk until a sync or close
is called. All posix operations, such as remove and rename, are atomic, even in event
of power-loss. Additionally, no file updates are actually commited to the
filesystem until sync or close is called on the file.
## Other notes ## Other notes
@ -119,20 +115,19 @@ All littlefs have the potential to return a negative error code. The errors
can be either one of those found in the `enum lfs_error` in [lfs.h](lfs.h), can be either one of those found in the `enum lfs_error` in [lfs.h](lfs.h),
or an error returned by the user's block device operations. or an error returned by the user's block device operations.
It should also be noted that the littlefs does not do anything to insure It should also be noted that the current implementation of littlefs doesn't
that the data written to disk is machine portable. It should be fine as really do anything to insure that the data written to disk is machine portable.
long as the machines involved share endianness and don't have really This is fine as long as all of the involved machines share endianness
strange padding requirements. If the question does come up, the littlefs (little-endian) and don't have strange padding requirements.
metadata should be stored on disk in little-endian format.
## Design ## Reference material
the littlefs was developed with the goal of learning more about filesystem [DESIGN.md](DESIGN.md) - DESIGN.md contains a fully detailed dive into how
design by tackling the relative unsolved problem of managing a robust littlefs actually works. I would encourage you to read it since the
filesystem resilient to power loss on devices with limited RAM and ROM. solutions and tradeoffs at work here are quite interesting.
More detail on the solutions and tradeoffs incorporated into this filesystem
can be found in [DESIGN.md](DESIGN.md). The specification for the layout [SPEC.md](SPEC.md) - SPEC.md contains the on-disk specification of littlefs
of the filesystem on disk can be found in [SPEC.md](SPEC.md). with all the nitty-gritty details. Can be useful for developing tooling.
## Testing ## Testing
@ -143,3 +138,19 @@ The tests assume a linux environment and can be started with make:
``` bash ``` bash
make test make test
``` ```
## Related projects
[mbed-littlefs](https://github.com/armmbed/mbed-littlefs) - The easiest way to
get started with littlefs is to jump into [mbed](https://os.mbed.com/), which
already has block device drivers for most forms of embedded storage. The
mbed-littlefs provides the mbed wrapper for littlefs.
[littlefs-fuse](https://github.com/geky/littlefs-fuse) - A [FUSE](https://github.com/libfuse/libfuse)
wrapper for littlefs. The project allows you to mount littlefs directly in a
Linux machine. Can be useful for debugging littlefs if you have an SD card
handy.
[littlefs-js](https://github.com/geky/littlefs-js) - A javascript wrapper for
littlefs. I'm not sure why you would want this, but it is handy for demos.
You can see it in action [here](http://littlefs.geky.net/demo.html).