[Unsafe][WIP] Using memmap'd data directly for CPU tensor storage, instead of copying.

[coreylowman]

Blocking questions:

1. Is it safe to std::mem::forget the tensor? Is that all we need to do? What about the other fields of tensor?
2. Is the Vec::from_raw_parts_mut usage safe?

[coreylowman]

Is this safe?

[sdake]

This is a cool feature, lazy loading via mmap() is how I'd default. There are some situations in which mmap() does not work (e.g. when the storage is exposed via a virtual block or virtual file system). I am certain that mmap() does not work with vhost devices that use virtiofsd.

It may also be that doing the work of loading once may produce better overall cache performance (inference would be faster, but initial model load would be slower). Definately worth benchmarking.

Perhaps make this optional?

Cheers,
-steve

[coreylowman]

Yeah I'm thinking to make this opt-in, because you're exactly right that you'd be paying for disk reads every forward call. Perhaps we can add something like --mmap-limit-for-model which defaults to 0

[sdake]

@coreylowman eventually the forward pass would warm up the cache as needed. I am trying to think of a use case where mmap() is superior here. The use of mmap() or not, depends on the holistic system. Either way, after the cache is warmed up, mmap() won't have much impact.

The advantage in my opinion is to mmap(), and then not lazily load, but force load the stuff that matters. That way you get instant memory mapping of the model, but can load the important stuff as needed, reducing what the user sees as "load-screen" lag.

[coreylowman]

Maybe use ManuallyDrop instead?