Cache protocol/data/experiment files

[kalekundert]

A lot of exmemo commands perform recursive globs to find files matching a keyword. These globs take longer and longer----eventually becoming a bottleneck---as the project grows and more files need to be searched. Part of the problem is that querying the filesystem is fairly expensive. I think I could speed these searches up significantly if I maintained a cache of the names of the files in the project.

However, a cache wouldn't do me any good if I need to update it every time I run a command anyways. (I definitely can't assume that new files won't be created between commands, and it would be annoying to the user if it were possible for the cache to be stale.) So I think I'd have to run a daemon to make this work. Having to manage a daemon would also be annoying, but I think I could make stay behind the scenes pretty well:

• When needing to convert a keyword to a path, check to see if a daemon is running.
• If not, start one. Use locks to prevent race conditions.
• Ask the daemon for the path. If the daemon just started, it'll have to check the filesystem first. So the answer will be right, but you won't get the performance boost. If the daemon has been running, it can monitor the filesystem via inotify and return an up-to-date answer immediately.