I'm a 3D technical artist and still lack some of the knowledge to fully understand this. As far as I understood, the main idea is to ensure that when you turn meshes or pointclouds into a machine learning model, you correct some of the measurement issues you have observed, right? In particular you avoid accidentally copying the original structure rather than teaching the algorithm the idea of the structure (that's overfitting, right?). I think i am missing how you use this result. What are some suggested applications? Did I see it right that you made a machine learning tool that extrapolates UVs. Or is the tool's purpose to remesh an object? If it's for remeshing, what application are you remeshing for? One thing I would like to see this expanded with, is a focus on key areas. As an artist I don't care for a blobby or even approximate version of a mesh that had clear features. I care for efficient preservation of detail. Like, if there is a tiny crease or a hard edge, I want that preserved and supported by the topology. Maybe that could be a follow-up? XD
Hi! I think you've got the right idea, but perhaps I can give some more explanation. Common learning tasks on 3D surface data include classification ("is this shape a chair/cat/airplane/etc?"), segmentation ('"label the vertices of this model as arms/legs/head/etc"), and correspondence ("match up corresponding locations on these two shapes so I can transfer texture data between them"). Methods for these problems like the one shown here are 'supervised', which means that we train them based on datasets of meshes for which we already know the answer. Then later we apply them to new, not-previously-seen meshes. It turns out when you train other existing neural nets to perform these kinds of tasks, they learn to predict based on the connectivity of the mesh; for instance if we trained on meshes that were all tessellated in one way, our neural net would output nonsense if we applied it to a new mesh tessellated with different vertex density, or regular vs. irregular edge flow. That's no good. Building these neural nets with DiffusionNet instead greatly alleviates this issue by design, because diffusion is a smooth, simulation-based operation. And since we already know how to do simulation in a way that doesn't depend on mesh too much (finite elements), our machine learning inherits the same properties. And agreed about using nicer, more detailed shapes as inputs! In graphics & vision research we are often guilty of using coarse, isotropically tessellated models that are easy for our algorithms to process. These are very different from the kind of meshes technical artists have found most effective. This is an extremely important bridge to cross, and we are working on it :)
@@nmwsharp Thank you for the clarification! I think even -or maybe especially- in the meshes you show, there is already a lot of lost information. All the little creases and crevices that make them look more like humans and less like shapeless blobs are important boundary regions and should be recognized by whatever algorithm you are developing. Even the panties the characters are wearing should be understood like separate objects on an underlying shape so a smart system can make good choices. For example if I did a UV unwrap I could use those boundary regions to make the seam as subtle as possible. I guess algorithms that deal with humans and animals should be trained in a special way anyways. When a trained artist looks at a human figure, they can guess at some of the bones and muscles underneath. I think if you teach the algorithm to see like that, they are less likely to make any mistakes. The fact that the algorithm learns about topology connectivity, means it could easily learn "anatomy" (because the connectivity of anatomy is mostly consistent across humans). This should also allow the machine to accurately predict animation skeletons etc. I believe there are systems that already do this, but I guess the key to improving those is to feed significantly more and better data to the machine. (Feel free to contact me on Discord: DASD#2225 if you wanna talk more and or maybe collab sometime.)
I'm a 3D technical artist and still lack some of the knowledge to fully understand this. As far as I understood, the main idea is to ensure that when you turn meshes or pointclouds into a machine learning model, you correct some of the measurement issues you have observed, right? In particular you avoid accidentally copying the original structure rather than teaching the algorithm the idea of the structure (that's overfitting, right?).
I think i am missing how you use this result. What are some suggested applications? Did I see it right that you made a machine learning tool that extrapolates UVs. Or is the tool's purpose to remesh an object? If it's for remeshing, what application are you remeshing for?
One thing I would like to see this expanded with, is a focus on key areas. As an artist I don't care for a blobby or even approximate version of a mesh that had clear features. I care for efficient preservation of detail. Like, if there is a tiny crease or a hard edge, I want that preserved and supported by the topology. Maybe that could be a follow-up? XD
Hi! I think you've got the right idea, but perhaps I can give some more explanation.
Common learning tasks on 3D surface data include classification ("is this shape a chair/cat/airplane/etc?"), segmentation ('"label the vertices of this model as arms/legs/head/etc"), and correspondence ("match up corresponding locations on these two shapes so I can transfer texture data between them").
Methods for these problems like the one shown here are 'supervised', which means that we train them based on datasets of meshes for which we already know the answer. Then later we apply them to new, not-previously-seen meshes. It turns out when you train other existing neural nets to perform these kinds of tasks, they learn to predict based on the connectivity of the mesh; for instance if we trained on meshes that were all tessellated in one way, our neural net would output nonsense if we applied it to a new mesh tessellated with different vertex density, or regular vs. irregular edge flow. That's no good.
Building these neural nets with DiffusionNet instead greatly alleviates this issue by design, because diffusion is a smooth, simulation-based operation. And since we already know how to do simulation in a way that doesn't depend on mesh too much (finite elements), our machine learning inherits the same properties.
And agreed about using nicer, more detailed shapes as inputs! In graphics & vision research we are often guilty of using coarse, isotropically tessellated models that are easy for our algorithms to process. These are very different from the kind of meshes technical artists have found most effective. This is an extremely important bridge to cross, and we are working on it :)
@@nmwsharp Thank you for the clarification! I think even -or maybe especially- in the meshes you show, there is already a lot of lost information. All the little creases and crevices that make them look more like humans and less like shapeless blobs are important boundary regions and should be recognized by whatever algorithm you are developing. Even the panties the characters are wearing should be understood like separate objects on an underlying shape so a smart system can make good choices. For example if I did a UV unwrap I could use those boundary regions to make the seam as subtle as possible.
I guess algorithms that deal with humans and animals should be trained in a special way anyways. When a trained artist looks at a human figure, they can guess at some of the bones and muscles underneath. I think if you teach the algorithm to see like that, they are less likely to make any mistakes. The fact that the algorithm learns about topology connectivity, means it could easily learn "anatomy" (because the connectivity of anatomy is mostly consistent across humans). This should also allow the machine to accurately predict animation skeletons etc. I believe there are systems that already do this, but I guess the key to improving those is to feed significantly more and better data to the machine. (Feel free to contact me on Discord: DASD#2225 if you wanna talk more and or maybe collab sometime.)
this is cool