WIP: Implement 'fuzzy skin' displacement mapping

[undingen]

This is a work in progress PR which modifies the fuzzy skin logic to displace the objects skin based on the values in an image.
Unfortunately I don't have a 3d printer so could only test this a bit in the slicer - I would love to see 3d print using it....

Here some sample objects with a brick displacement map applied:

To enable: set "fuzzy skin noise type" to "Displacement Map" and change "fuzzy skin displacement map filepath" to the filepath to the image to use. The image must be a small (best to be around 128x128px) png grayscale 8bit per pixel displacement map. Each "fuzzy skin point distance" is on pixel in the texture (adjust it to something like 0.1 to get more detail).

It's based on the initial work of @Poikilos in PrusaSlicer which unfortunately did not gain much interest prusa3d/PrusaSlicer#8649.
There have been various feature requests for similar things: #3992 #5229

This is of course not ready for merging but I would like to see if the community is interested in this feature and get some feeback from the main contributors on where to store the png. There seems to be no existing feature which has the same requirements.
Also I would appreciate if @Poikilos is looking at it, I'm not sure if its my modifications or a problem in the additional code but I do see some artifacts on objects depending on orientation. If a wall of the object is closer to 45° orientation it starts to show up - worse with arachne wall generation - seems to be some rounding errors/float precision?.

I integrated with the #7678 as an additional noise type. But if this patch is not getting merged I can modify it to work without it. So please only take a look at the last commit implemented by me.

[OskarLindgren]

I don't have a 3d printer so could only test this a bit in the slicer - I would love to see 3d print using it....

I have both an ender 5 pro and a p1s, so both ends of the spectrum; I would love to help you test this in the real world (especially with multi material to see how that effects things) to see if it's viable and what issues could arise, please reach out to me either here or on discord: divide05

[discip]

FINALY! 😊

Awesome! 👍

[discip]

First of all: Thank you very much for this PR! 👍

I think, for consistency reasons, it would be nice to either have all displacement maps in a predefined list (Perlin, Billow, Brick ...), or to have to select them by specifying a path.

Or even better:
A separate folder in which displacement maps are stored and their respective names are listed in a drop-down menu under Fuzzy Skin Noise Type.

Please correct me if I'm wrong. 😊

[Keavon]

I'm not familiar with the code so I can't assess how easy or hard its implementation would be, but from a product design perspective, how about a drag-and-drop experience where the texture file can be placed onto the 3D model directly, or onto the dropdown menu in the UI which is used to pick the texture? It would probably be sufficient to just store one at a time (whichever was dropped onto the model or dropdown menu most recently). The dropdown menu could say "Custom Texture" if one is active. I think the drag-and-drop experience is reasonably easy but there could also be a dropdown entry called "Browse for Texture..." which, upon being selected, immediately opens the OS file browser dialog, and once successfully selected, switches to "Custom Texture". This would be preferable to putting files into some magical directory that users need to locate.

[Keavon]

The image must be a small (best to be around 128x128px) png grayscale 8bit per pixel displacement map. Each "fuzzy skin point distance" is on pixel in the texture (adjust it to something like 0.1 to get more detail).

Question: how does this all work to map a 2D texture to a 3D model if not with a UV map or some other technique like triplanar projection? And just to bring up the idea, how feasible would someday extending this to support UV-mapped displacement textures for full control of micro details on a model?

[Poikilos]

The way to avoid artifacts in angled walls is either (depending on the shape of object, one or the other is better):

• use the object's rotation to transform the cube projection (vector math is not my strong suit, I may need some help here)
• add cylinder projection (the math is pretty easy here, but it works better on objects with a curved top profile such as an upright cylindrical tower, not as well for sides that are flat from top view)

[Poikilos]

The image must be a small (best to be around 128x128px) png grayscale 8bit per pixel displacement map. Each "fuzzy skin point distance" is on pixel in the texture (adjust it to something like 0.1 to get more detail).

Question: how does this all work to map a 2D texture to a 3D model if not with a UV map or some other technique like triplanar projection?

Look up "Cube Mapping" online to get the idea of it. Primitive shapes were often used as an alternative to UV mapping in early graphics. Primitive mapping is still often used for environment maps of various sorts (backgrounds, fake [cubic] reflection maps).

• Cube mapping is mostly a lost skill (except can probably be exported from Blender) so I think the 4-sided partial cube map is good for people who just have some seamless rectangular texture and want to throw it on a model. In fact, you can create a cube map that maps to the model perfectly, such as projecting an unwrapped UV map to a cube UV map in Blender, if you arrange the cube the same way (only 4 sides are used by my implementation, so just squish the top and bottom flat, and map the 4 sides to the shape of the texture as a square/rectangle).
• Of course, with models such as characters, there will be issues with concave areas including "very concave" areas such as between the body and an arm since the same part of the UV will cover both. Cube mapping (or my 4-sided cube mapping) is more suitable for structures, but could look ok on an organic model.

And just to bring up the idea, how feasible would someday extending this to support UV-mapped displacement textures for full control of micro details on a model?

There seem to be 2 options, and option B would be probably better (way easier, but still requires some work in regard to code usually only in graphics-oriented 3D programs), and B would reside in entirely different code to this PR:

A. (hacking this code even further, as in sending even more data to the slicer, not advisable)
You have to know the z position of the layer and the 3 verts to make a calculated normal (not a stored normal if present) of the face to recover the point's position on the original face using raycasting:

• The face and face index (as corresponds to the UV map) would have to be attached (as variables/pointers) to the sliced point or segment resulting from it, then passed along to each fuzzy point (as a vector of 3 3d points and 1 face index or pointers to those 2 things).
• During the displacement step, that is the hard part. You have to do raycasting from the fuzzy point to the preserved face, at a straight angle to the face:
  • calculate the face normal
  • move the vector along the face so it points to the position of the fuzzy point
  • intersect the ray with the face (raycasting) to recover the triangulated position
  • go back and recover the UV from the UV mapped face with the preserved face index, using that triangulation data to do the actual UV mapping

In other words, it is not very feasible, but possible.

B. (simpler, more standard)
The only way I could see around A is to utilize a UV map before slicing (before course segment is sent, which in this case would become detailed and bumpy right away), which would make the feature entirely separate from fuzzy skin which is done after (but technically compatible, if you want extra bumpiness overlayed on the UV mapped displacement). You'd probably have to cache a detailed version of the model with displacement applied, at minimum to the detail of the image, or maybe adjustable to higher for smoothly sampling the image. Essentially it is traditional displacement mapping in that scenario. Just send the detailed cached model to the slicing algorithm and you're done.

If B were done eventually, that would replace code in this PR. It is something I didn't think of earlier. But it is similarly tricky in that there has to be a cached model with all of the detail applied to it, and it requires more modeling code (hence significantly more code added than this PR, depending on what is available in your 3D engine).

Also, keep in mind STL does not have a UV map. That and the complexity are reasons why I didn't do it. If we at least get primitive mapping working well for various object shapes that is probably going to solve most use cases. However, if someone knows the vector math better than me and can do what I described above with raycasting, or the preferred pre-slicing displacement (option B), go for it. If we get that working, people may start to use other formats such as OBJ which support UV maps. Probably save UV maps for later, to keep an incremental approach.

[daniiooo]

This feature looks great. I would like to try it out but I'm having hard time procuring decently working displacement maps.
Would you mind providing some to test it with?

[Poikilos]

Displacement maps:

• https://opengameart.org/content/hand-painted-texture-sandstone CC-BY 3.0
  • Sandstone_DISP.png
• https://opengameart.org/content/brick-wall CC-BY 3.0
  • brick_wall2-disp-512.png
• https://opengameart.org/content/brick-2 CC0
  • Brick_2_height.png (in zip)
• https://opengameart.org/content/handpainted-brick-texture-pack CC0
  • ruin_wall_03_height_0.png

[daniiooo]

Alright I got one test print going with brick wall since this one worked out of the box. Sandstone didn't show up on sliced model with the same settings.

Settings

I'm not sure if those walls should be considered overhangs so that might be something that'll need ironing out.

Sliced model

Another potentially unwanted behaviour is that when using all walls setting for fuzzy skin even the interior walls are being displaced. Thats not the case for contour or contour and hole. This affects very heavily places where only walls are present like in photo 4.

Contour

All walls

Here are the photos of test model.

Photo 1

Photo 2

Photo 3

Photo 4

From other miscellaneous obervations, like fellow users mentioned there should be an easier way to apply the desired map, something like drag and drop or a file picker option.
Another useful option but nothing really critical would be some way to display the map on the not yet sliced model to see how the settings affect it.
Also the fuzzy skin filepath isn't getting saved properly in the 3mf project file on windows. After saving and reopening the project the backslashes get removed.
For example a path C:Users\user\Desktop\test_1.png becomes C:UsersuserDesktoptest_1.png

For reference here is the 3mf file and gcode of the print.
displace_map_test.zip

[undingen]

Thanks for all the input! It's encouraging to see interest in this PR.

In my opinion, adding proper UV maps or significantly increasing mapping complexity isn't the right approach. Even if we were to support UV maps, it’s likely that anyone needing such precise application would also want features like bumps on horizontal surfaces - something that isn’t feasible with the (current) fuzzy skin logic.

I believe users requiring this level of detail would benefit more from modifying the model's geometry in Blender or a similar program, which is far better suited for the task than trying to reimplement such functionality here.

If this small feature set ends up not being practical for real prints, at least the prototype serves as a useful exercise in highlighting that changing the geometry might be the better solution.

[undingen]

Alright I got one test print going with brick wall since this one worked out of the box. Sandstone didn't show up on sliced model with the same settings.

Thanks for your detailed report and uncovering new bugs!

There is a bug I did not spot: depending on if wall generation is arachne or classic the bump map is applied negated. You tried with arachne which made the cement between the bricks dent out and the bricks dented in. In classic its the opposite. Will fix this later today. This likely is also the reason why I said in the initial post that arachne seems to have more issues - I likely did not notice that the displacement map was applied swapped...

Concerning the overhangs:
@OskarLindgren (thanks!) and me experimented with it a bit more (final results are not in yet) and we noticed that the bricks displacement map we where using had way to big steps between the cement and brick part which made the overhangs too big - likely adding a feature/option to smooth over the texture to create intermediate steps would be helpful to easily avoid running into to big overhang if they use a stock displacement map from the internet.
Also fuzzy skin thickness of 0.6 is quite a lot - maybe sticking around the 0.3 is better for now.
The other issue is the artifacts generated on the walls I will have to investigate more why they are here: there are horizontal lines in the sliced code on flat bricks which should not really cause an issue.

I will also add a workaround for the filepath issue in windows.

This is one of the benchys of @OskarLindgren which shows the problem with to big overhangs when the texture is switching between black and white instantly without making steps...

[undingen]

I pushed three new commits:

• make the displacements go both ways like they do for the random noise ones. Before I only return values between "0 and fuzzy_skin_thickness" not its "-fuzzy_skin_thickness to + fuzzy_skin_thickness". This is doubling the selected fuzzy skin thickness so configured values need to be smaller now.
• arachne applied the offset swapped. This seems not related to my patch but before was not noticeable because it was just random offsets...
• reordered the menu items and hide some unused ones

Concerning the windows file path \ issue: this seems to be a problem with the config save/load code which does not correctly escape the string (double unescapes it thereby removing the \). Could you please as a workaround manually changing the \ to / - I think they should be supported by Windows and will not cause issues when storing.

I played around with slightly rotating a cube and I can definitely notice that this changes how the displacement map is applied and that it does not take into account the rotation of the object: I think all coordinates are relative to the bed and if the object is slightly rotated x/y will only marginally increase but the code expects it to increase by a full step.
The cube in the distance is slightly rotated the front one is aligned to the bed axis:

[undingen]

@daniiooo
With the new code (which correctly applies the displacement map and not inverted) I think its slices with less overhangs (I did reduced skin thickness to 0.3 but I think this should be like the old 0.6 you used):

[daniiooo]

The artifacts look way more pronounced on low layer heights(0.08mm in the attached picture)
Point distance: 0.08
Thickness: 0.3

Example

The workaround for path not saving properly works. Since it's a config save/load code though it probably should be fixed anyway even if not in this PR. If there is anything specific that needs to be tested out with actual prints feel free to reach out.