I want to figure out a good local-first way to sync files between my devices and the servers I publish to. I am documenting my efforts toward that goal here.
I think I'm done with the first drafts of the tools making up the stack comprising my datastore system. I guess it's time to start really testing things out in the wild. I have a decent backup regimen, so I'm not too worried about data loss...
The tools and their functions, starting from the lowest in the stack:
This exercise has changed and challenged my perspective on some things. I've started thinking more seriously about what a path-free filesystem might look like, or how data might be organized in a more unorthodox fashion. I've begun considering splitting lsd into two parts, since I think the allocation of filespace and management of permissions can be separated from notions of file paths. If I ever act on this, it'll likely be years in the future. I want to get back to my food bot project as quickly as possible. This diversion took a lot more time than I'd expected.
I was able to quickly knock out a first draft of a tagging and tag searching utility, which I'm calling lyts. Now I'm all the way down to the last two bullet points in my list of datastore goals, which means I'm having to think about how I should structure my markup in order to derive decent representations in HTML and Gemini markup. Since I also want to give screenreaders a decent experience, I've begun playing around a bit with orca and Firefox. I've restructured this page a bit in response to what I've learned so far, and I'm considering whether simply using more descriptive inline link text and then pulling those out into Gemini links at the end of every paragraph might be the best solution for me.
The simplest way to deal with this would be to just write everything in Gemini markup and render it to HTML, but I'd really prefer to find an approach that produces something that feels native to both the Web and Gemini.
Rewrote LSD to operate based on face/data pairs, with face files holding all attribute, extended attribute, permission, and path data, and data files holding the actual file contents. This is not my preferred design, as I've previously said, but it's working a lot better than the ones I've tried before. Now I'm working on cleaning up code and designs left over from previous iterations. I'm sure I'll find plenty of bugs, but at least Syncthing is happy now.
Thought about replacing LSD's extended attributes with face file contents, to make what I'm doing more obvious to anyone who might stumble upon a backup of my data, but directories are technically "face files" as well, so this wouldn't work. Instead maybe I'll have LSD create a README explaining everything in the root of its data directory upon setup. I really don't want to wind up with a mess that I won't be able to recover from when/if this system fades into the background of my everyday life and I inevitably forget all the implementation details, or if I die and someone less technical winds up having to pick through all my over-engineered nonsense.
Keeping all this in the "simplest" branch on Codeberg until I'm confident that I've got it mostly right this time.
Distributed file synchronization is a really hard problem and I really don't want to jettison Syncthing because then I would have to deal with that problem directly. This whole thing was supposed to be just a quick and dirty hack anyway. Simple, at any rate. Conceptually, it is. Practically, I keep running into problems. I think the biggest source of complexity right now is how LSD's FUSE overlay interacts with Syncthing. I'm getting file conflicts every time I change a file for some reason. I could dig into Syncthing some more and figure out why it thinks there's a conflict. This would likely allow me to achieve my goal with minimal additional code, provided no additonal complexity is lurking in the interaction between the two. (I'm skeptical of that; hidden complexity seems like the rule here.) But it's also appealing because figuring this out would likely improve my FUSE layer. Most likely there is something wrong with it that isn't apparent to an obtuse end user like myself, and which could crop up in more subtle ways later.
On the other hand, getting rid of the need to keep face and data files in sync would let me at least get rid of the FUSE layer between Syncthing and the data it's operating on, which would save me from having to concern myself so much with Syncthing's behavior. I could achieve this by making each nodeset a flat repository of GUID-named files with permissions set so that only the user running the daemon can modify or read them directly, which would route around the very real issue of data being exposed when the permissions on the face file suggest otherwise. I would still need a FUSE layer for LUFS, to deal with directing file queries to the correct nodeset and to avoid having symlinks everywhere, but I might be able to generalize the code to deal with other protocols as well. E.g., if I wanted to "reduce" a file that could be accessed over HTTP, maybe this could be conceptually merged with "reducing" an LSD-stored file.
I did really want to avoid the "folder stuffed with inscrutably-named files" thing, though. Also, this design choice would mean I'd have to run some sort of "garbage collection" on the data store in order to remove anything which no longer has a reference to it. I don't like that so much.
Syncthing's "watch" function also seems to not be working with the overlay, which makes me think my overlay doesn't work with inotify either. It's all very screwy. I should welcome the chance to improve the FUSE layer, I guess, since there's no clear way to get rid of it entirely. I'm just getting really tired of this part of the project.
I'm starting to feel like I'm doing a lot more testing of how Syncthing works and trying to work around it than anything else. It's making me think that I maybe should just roll my own synchronization layer that I'll at least understand.
I just reduced a file on a node other than the one I was running the command from, which is supposed to be the simple case. It just moves a data file from one Syncthing folder to another. It did that, and then Syncthing restored the file to the original folder as well! So the situation was meant to be "A/sidecars/file", "B/data/file", where "A" and "B" are different nodesets, but it ended up being "A/sidecars/file", "A/data/file", "B/data/file". Perhaps the issue is that the extended attributes were updated on the file before the move. Might be that Syncthing is so adverse to data loss that it doesn't delete a file if it was deleted before changes propagated or something. At any rate, I really dislike how much Syncthing's behavior has begun to intrude into my thinking about the design of this thing.
Maybe I'll try the "directory stuffed full of inscrutably named files" approach before I ditch Syncthing altogether.
After thinking about it some more and sleeping on it, I think I'm going to try abandoning the "simple" .stignore-based reduction scheme and try using xattrs as a signaling mechanism. I think by declaring the nodeset a sidecar or data file should belong to and then making it so in the Setxattr function, I might be able to get reduction, restoration, and (probably) exclusion working as I'd originally envisioned. Inclusion will still need to be done from a node that already has the file, but I think this won't bother me too much.
This should allow me to avoid the GUID-based filename situation described in yesterday's post and continue with the "parallel paths" scheme I've been working with. This will also keep Syncthing in charge of handling conflicts and ordering, which saves me a headache. I don't see anything obviously wrong with the idea, but I won't really know until I try it. Shouldn't take too much work to try, at least not in comparison to the direction shift I was considering yesterday.
I've gotta get better at keeping track of why I made certain design decisions. I'd avoided using .stignores for reduction because doing so then means I'm relying on editing that file every time a reduced file gets moved or renamed. The thought of doing things this way gives me a bad feeling in my gut, and if I unpack that feeling I come up with these objections:
I had really wanted to avoid making a directory stuffed full of inscrutably named files part of my design, but I'm starting to feel like maybe this is my best option, given the alternatives. While returning to the messaging-over-Syncthing scheme would allow me to avoid such a thing, it also introduces some of the sort of complexity I'd hoped to avoid by using Syncthing. Like, if I'm going to have to use CRDTs for messages anyway, I might as well go whole-hog and use them for the whole filesystem, right? The problems just seem similarly complex, and I'm really trying to avoid that kind of complexity.
What I'm thinking of doing now: keep all filenames and directory structure in the "sidecars" directories, but make the data directories flat repositories of files named by their LSD GUIDs. Then use the LSD layer to make operations on sidecar files transparently pass-through to their respective data files based on their "user.lsd.id" xattr values. I've seen a couple people and projects using a similar scheme, but with symlinks instead. While that's more transparent with regard to how the system is constructed, it also means that any naïve copy action will wind up with nothing but symlinks to nowhere. Also, I've used git-annex before and, while I don't remember much from that time, I do remember really disliking the output of "ls -l" with everything being a symlink. A file being a symlink carries some amount of contexually-informed semantic weight for me, and if everything is a symlink then that information just gets lost in the noise.
I think this design nicely resolves the tension between a file's identity and its name by unifying identity and name on the data side and then using the zero-byte "sidecar" files as mere labels pointing to those identities. This allows those labels to be easily moved around and renamed without consequence for their respective data files. It makes sense to me why this design pattern seems to appear so often in this problem space. I'd just really hoped to come up with something with a file structure more easily understood by people with no knowledge of the system which produced it.
I'll still have to modify .stignore files to reduce data paths (in order to delete paths without propagating the deletion when a node reduces a path on itself), but at least I won't have to mess with them every time I want to move or rename a reduced file.
I may also be able to reintroduce the ability to reduce, restore, exclude, and include files for any node, as in the pre-simplified design, by using xattrs themselves as messaging mechanisms. I'll consider that option more thoroughly once I've cleared this hurdle, since my energy levels and available time are significant factors here.
Edit: No, this is not an April Fools joke.
Edit: Shoot. Having the LSD layer transparently passthrough file operations on the "name" files to the underlying data files will play merry havoc with Syncthing, particularly when reduced files are involved. The easiest fix would seem to be using symlinks. I don't like this. 🙃
I may be able to reasonably stop keeping file attributes synced between name and data files under this scheme, which would allow me to point Syncthing at the "sets" directory instead, only using the LSD mount to provide passthrough operations to LUFS. 🤔
There were indeed bugs, but I've got the most obvious ones ironed out. Still trying to get consistent syncing, though. Syncthing seems to not be noticing when a file moves between nodeset directories for some reason, at least not until it does a full rescan. It's also presently having trouble syncing a data file to my laptop from my server after including the laptop, despite syncing the sidecar file just fine. No error messages, nothing obviously broken, but still: not transferring the 13 bytes comprising the data file.
I do have a server with a public IP address, and the temptation to use it to switch to something a little less "magical" for syncing, like Pijul or Git, is fairly strong. Neither seem to support xattrs, though. Since I am already using sidecar files, I could just shove metadata in those, but... I really don't want to loosen the bond between a file and its metadata that much! I'd also have to figure out a good way to deal with binary files, since neither play well with those out right of the box. Plus I don't care so much about file history, just syncing, which adds to the feeling that Git and Pijul would simply be poor fits.
I really would just like for something to work consistently is all. Hopefully the trouble I'm experiencing with Syncthing at the moment is owing to something I've overlooked, and hopefully my error will become fairly obvious soon. I've not generally had this sort of trouble with Syncthing before, but I've also never used it this heavily before.
Edit: This log seems to be my "rubber duck" or something. Not even ten minutes after writing this and I'm now seeing an "insufficient space" error in Syncthing's logs on my laptop. There is indeed sufficient space, so something must still be messed up about my LSD overlay.
Edit 2: There were two go-fuse interfaces I'd not yet implemented. Implementing those and making a few other small changes seems to have fixed everything. Extended attribute and data file syncing now seem to be working just fine. I'll be moving on to reduction and restoration testing tomorrow, now that moving files between nodesets seems to be working.
Just completed a "first pass" on the simplified version of LSD. Code's written and compiles. I don't think I missed anything, but I expect it'll break a bunch because I haven't actually tested it yet! Still, I'm glad that I managed to get at least a first draft completed. This isn't normally how I like to write things, but it's an easy pattern to fall into when I haven't quite got a clear picture of how things should fit together yet. I guess you might call this a prototype. I learned more about this problem and have a clearer picture now of how things might fit together.
I think if I do a second version, I might try cutting out Syncthing. There's a paper I haven't read yet on a project called ElmerFS that seems promising. If this ends up working well enough for my purposes, though, there may not ever be a second version, which I suppose is well enough. This experience has got me thinking more about alternatives to traditional file systems anyway, so that might be the next experiment once my data organization scheme starts feeling insufficient again.
I played with git-annex today and found it to be too unreliable and complicated. On paper it certainly sounded workable, but the "git annex assistant" command too often did not sync changes to the USB stick I was testing things with. I found files in my repos had their contents sometimes replaced with git-annex's internal path for the file as well. I also once somehow managed to trigger a conflict that I could not figure out how to resolve. Finally, the "preferred-content" functionality I'd hoped to leverage to create "reduced" files didn't ultimately map cleanly to my mental model, which revolves around moving paths between sets of nodes.
After reaching out for an update on Bazil's status¹ from Tommi Virtanen (the author) and finding myself running out of steam as well, I'm going to give git-annex a go. I'd tried it years ago and abandoned it, but I can't remember why now. On paper it sounds close enough to what I want, with the advantages of not taking up so much of my limited spare time to get going and also being fairly mature as a project. I'll report back if it works out.
¹An excerpt from the author's reply: "The design for Bazil ran into a funky edge case with version vectors vs whole-subtree renames that I never gathered sufficient motication to think hard enough about to figure out an answer. It's complex enough that I think the only reasonable next step would be to start formally proving the sync algorithm."
I've decided to call my syncing daemon "Lykso's Syncing Daemon," in keeping with the naming convention of LUFS ("Lykso's Union File System"). The work that needed to be done to finish the things I'd overlooked in LUFS went very quickly, so now I'm back to working on LSD. My work with LUFS should help the work go smoothly, but the trick will be (as ever) carving out time to get it done. My wife has been very understanding and took more than her fair share of the baby-tending duties so I could wrap up LUFS, and now I've got to make up for it.
The changes in the syncing daemon setup meant I could remove some code from LUFS, which I did. In working on this, I discovered my LUFS implementation was not as complete as I'd thought. "Renaming" (i.e., moving) files required I implement an interface I'd previously overlooked, and I discovered that there were similar interfaces I'd overlooked for creating directories, listing extended attributes, setting extended attributes, and getting extended attributes. I've finished implementing the interfaces for moving files and creating directories and will shortly begin work on the extended-attribute-related interfaces.
I'm of the mind to create a notion of sidecar layers in LUFS, and have in fact done most of the required work so far, but I'm not sure whether it actually makes sense to introduce this notion at that layer upon further reflection. I'd hoped to avoid having to monitor the filesystem for changes, but I don't know if that's actually something I can work around here...
Addendum: I've given this some thought and I think I've worked out a way to keep sidecar files and main files synced without having to resort to filesystem monitoring, at the cost of one additional mount point. I've also determined that LUFS is absolutely the wrong place to be managing sidecar files given that I want changes to underlying layers to flow over into other layers, without having to go through the user's mountpoint. Going through the user's mountpoint does not work for Syncthing-initiated changes, which is what a change propagated to a "main" file from a "sidecar" file on another node, one without the "main" file, amounts to. I'm going to remove the sidecar code from LUFS, continue implementing the missing interfaces, and then work on introducing the necessary sidecar code to the syncing daemon. Syncthing will be syncing through a FUSE mount created by the syncing daemon, allowing the daemon to catch and respond to all file events as they happen. In theory this will work as a cross-platform alternative to watching for file events and as a resource-light alternative to polling the directory tree for changes. (I'm not entirely committed to making this thing cross-platform, but I'm going to try to at least avoid putting up roadblocks to that goal.)
Finished the setup code a few days ago. Was a lot simpler than I'd thought it would be. Not all the nodes have to be in setup mode, as I'd feared would be the case. Syncthing, it turns out, handles shared folder identity a lot better than I'd thought it would. Shared folders are considered the same if they have the same ID, which is set by the user, not by Syncthing. So all I had to do was slap an "lsd-" prefix on the SHA256 hash of the sorted list of node GUIDs for each nodeset folder and I had a folder ID which would be calculated the same way by each node.
Now I'm trying to work out how to deal with messages and conflicts. The "correct" way would be to use CRDTs to resolve conflicts, but I'm not sure I want to go that hard. There's a whole research project in there, and I already have one stalled project, maybe two, that would qualify as that. I just want something that works well enough for now. So I think my decision is to just let things conflict. The conflicts not already handled by Syncthing are just exclusion and reduction conflicts. I think the way they'll play out without global synchronization works well enough for my use case. Thinking it through as I write, here are the cases that are apparent to me:
There are more cases once we start digging into reduction, but I'm tired of typing all this out. The fact that this is a prototype meant to be operated by a single person with a more or less consistent understanding of their filesystem will hopefully be my saving grace here. The truly correct way to do all of this would be to ditch Syncthing and build up my own syncing scheme based on ElmerFS or similar. Or else I should join Bazil's efforts, given the years of work they've put into basically this exact problem...
Unedited stream-of-consciousness ahead:
As I consider this problem more, maybe my trouble is in the notion of nodes trying to exclude data from other nodes. This seems like a fundamentally flawed notion, like trying to turn back the hands of time. Clawing back information once shared. Perhaps nodes should only be able to exclude themselves, and the notion where other nodes are concerned changed to something more along the lines of requesting that a node exclude itself. Perhaps, given my reasons for writing this whole thing in the first place, I could do away with the notion of trying to "take back" files once shared altogether.
I suppose there's a fair notion of excluding other nodes from future changes to a file, which is what it seems like not attempting to resolve exclusion and reduction conflicts amounts to. Maybe this is the mental model I should try to employ.
Made some progress, largely on the "day off" my wife gave me a couple days ago. (We've discussed each having a 24 hour period once a month where one of us takes care of the baby and the other can go be baby-free for that time, and I've just had the first of these.) I set up at a sort of beer hall with food carts around and got my head on straight about how to proceed with the setup step. Wrote a bit of code as well, but ran out of time before I could get everything done. (I did do other things as well; this did not take the whole 24 hours!)
I'll share the code once the functionality has caught up to the speculative README I wrote to organize my thoughts.
Not much progress lately. Partly due to the holidays, partly due to a mental block I have around doing anything with Docker again. I used to use Docker all the time for things, but it's been a while, and my last contact with it involved trying to get it to stop overriding my iptables rules, which led me to feel that it's a bit of a sprawling, batteries-included mess. So I've been put off using it, but I also am unsure how else to automate testing. I may just... do it live. This is a bit of a toy project, and I'll be keeping backups of everything anyhow. This notion also irks me, though, because I've been bitten by that approach to things more times than I can remember.
I think the solution proposed in my last entry will work. I've settled on two verb pairs for now: exclude/include and reduce/restore. So there will have to be "reduced-by" and "excluded-by" tags.
Getting the initial folder and device synchronization going will be bothersome. I'd wanted to lazily create "node set" directories as they were needed, but I think I'm going to have to have a "setup" command that creates all the possible combinations and shares them all at once. So any time I want to add a node, all the nodes will have to be in "setup" mode and connected to each other, which... is not ideal. I don't think there is a way around it without replacing Syncthing with my own code, though.
Each node set will choose the member node with the "lowest" GUID as the "leader," meaning it will be the node responsible for creating a directory for the node set locally and then creating bidirectional shares with each node.
I think this is going to have to be a functioning prototype, with a proper implementation taking control of the synchronization layer as well. Hacking together something based on ElmerFS seems like it might be a viable route forward, should I still have the energy to do this properly once the prototype is done.
If I don't, I suppose I'll at least have something usable, if a bit hackish.
I've just realized a node can request files it should not have access to and there is no reliable way to prevent this in my current design. As I've started writing out possibilities here, I think my best option might be to write an "excluded-by" tag with a reference to the GUID of node that excluded the file, to be removed when the file is restored, and without which the file cannot be restored by the requesting node.
The first piece is more or less ready. Lykso's Union Filesystem is a FUSE-based union filesystem with a couple unusual customizations to work a bit better for my requirements than the more common unionfs-fuse would. First, if there is an unlink followed within one second by a create at the same path, the new file is created in the same directory tree the old file was deleted from, as it would be with a simple write. This ensures that programs, such as vim, that unlink a file and then create a new file at the same path when saving don't also end up moving that file from the lower directory tree it originally resided in to the top one. Second, when a directory is unlinked, it deletes everything at the same path across all the mounted directory trees. This is to avoid ending up with a bunch of empty directories across the mounted directory trees after an rm -rf.
There may be bugs lurking in its corners and there may be further improvements to make, but so far it seems to do what I need it to do.
I've spent the past couple days weighing my options, arriving a few minutes ago at what I think may be my best shot at achieving these first two of these goals with minimal programming on my part.
Let there be a set of directories, one for each unordered combination of devices n through 1, where n is the number of devices. Each directory is synchronized via Syncthing with each device referenced in each directory's set of devices. A device in a set may choose to exclude a file from another device in that set by moving the file to the directory representing the set without the excluded device. It may likewise reduce a file to its metadata (e.g., external references) by first excluding the file and then putting a metadata file in the file's old location. (N.B.: I may instead make a separate directory tree for these files to make it easier to discover reduced files on a device.) Restoration is achieved by removing the metadata file and returning the original file to its original set.
Let there be a separate directory for messages between devices, synchronized between all the devices. A device may request a file's exclusion from, reduction in, or restoration to their sets by creating a message file containing a signed request. Message files shall be named via hybrid logical clock and processed in order. When the message file is received by the other devices, a daemon running on the other devices checks the signature and honors the request, deleting the message file after.
Metadata files representing "reduced" files are "sidecars," which I had hoped to avoid, but they do not appear to be avoidable, given the "reduce and restore" mechanism, without writing a custom file synchronization layer to replace Syncthing. The task of keeping the locations and properties of these sidecars in sync with their files will likely fall to the message processing daemon. Unique IDs will also be assigned, via extended attribute, to each file/sidecar pair to assist in repairing file/sidecar synchronization failures.
A device's set of directories are layered atop each other via a custom FUSE-based union filesystem that ensures each file remains in its originating directory tree when modified or moved, providing a unified view into the structure and allowing each device to have a different "default sync set" for files created locally. A special utility will be provided to make it easy to exclude, reduce, or restore files and to view any responses (e.g., error messages) originating from requests made via the message-passing directory.
.stignore files may of course still be used by nodes to exclude certain files, but this scheme is meant to avoid accidentally syncing large files outside of ignored paths, to avoid having to write utilities to manage .stignores, and to keep the .stignore files from possibly growing unmanageably long. It also has the advantage of providing a high degree of granularity for "source" devices regarding which other devices each file or directory structure should be synced to. Finally, the message-based reduction mechanism allows for the prevention of accidentally reducing away the last copy of a file.