That sounds awesome! – I’m dealing with this clothes skinning problem right now actually.
Yep, the up directions should probably be done in something like an outliner/hierarchy area. That is, maybe provide a local/world space toggle for the generated bones. Base this up direction on bone chains (rather than the individual bones themselves).
Here are some facts about bones generated this way:
-
Bones are defined in the outliner via color.
-
Bones are arranged in the hierarchy based on order of selection in a multi-selection list.
-
Bone centers are defined based on center of a volume of points for all verts colored with a particular bone ID.
-
A rough start/end point is generated for each bone based on hierarchy of color ID or using chain/hierarchy data if necessary (see next bullet point).
-
Special bones (i.e. start/end bones in a chain) can have their start or end positions offset locally to determine the “start” of the first bone or “tip/end” of the final bone when they are the first or last in a set of bones defined as a chain.
-
Bone that are offset as a “start/end” point are used automatically to determine the direction/location of all of the bones in that particular chain hierarchy.
-
Bone lengths are determined by vertex positions relative to their chain direction (if they are part of a chain), otherwise the direction can be tweaked by rotating it around
-
Bone length for individual bones is always automatically determined by the last vertex in the volume of the current bone ID’s direction/orientation.
-
Bones part of the same chain will have their start/end bone positions automatically determined by their hierarchy and start/end points.
Hopefully that gives you an idea of how I’m thinking this might work.
Honestly, everytime I think about it, the very fact that I have to build the rig first has always seemed like a waste of time to me since the “bones” concept is strictly user-facing – the game engine only cares about the weighting of individual verts in the end, so having to create bones is only a convenience if you plan to later attach something to the model or need some control points to drag around to pose it and keep track of the vert deformations.
Ultimately, selecting multiple bones and dragging them around (or applying a hierarchy sorting function to them) in an outliner to determine bone hierarchy after painting bone IDs on the mesh (that determine which verts the bones would influence) would be much easier than manually fiddling with rotation, scale, parenting, or painting weights for tiny bones (such as the fingers) by hand. An outliner of bone names/colors that act as their ID is really the core to this. Just use the selection order of the “bone” names in a list of multi-selected bones to arrange the generated “bones” into a hierarchy (i.e. simply RMB → “Arrange in Hierarchy” in that outliner). The bone origin could be determined based on the center of a point volume + offset (if you wanted to tweak an individual bone center/start/end point manually), then bone direction and length could be easily be derived from the left or rightmost verts of a “start” bone in the outliner hierarchy and pointed to the “tip” of an end bone in the outliner hierarchy. These would be assumed to be positioned in space relative to one another (the last selected bone’s potential “tip” would determine the end point direction of all the center points of the individual bone ID point volumes, and the first selected bone’s “start” point (taking into account its offset) would determine the overall starting point of all the bones in the chain, then push this point out toward the next “center” of the nearest bone(s), and adjust the length to the overall paint job for each section (maybe using “rings” of vertices to help guide you if necessary, but otherwise the manual offset of a bone’s “start” could be used for things like curved fingers, for example). Something like this would allow you to get a general bone chain “positioned, parented, and pointed” without having to the tedious process of fiddling with creating the bones themselves – you could simply paint their IDs on the points and have your rig be generated based on clouds of points you’ve defined by painting them (which you’ve already got), their center volumes (volumes determined by vertex color / bone ID), and the relationships between each of those volumes’ points to determine bone length and direction.
I would consider adding one extra layer (i.e. alpha channel) to use black/white to determine the amount of blend to be applied to each of the nearby verts weights for mixing/fading influence (so that the colors themselves would be straight 100% the RGB value you applied for each vertex) so you won’t have to fool with selecting which bone to blend with which other bone - as the alpha channel applied to that vertex would determine this blending amount – the color channels themselves would simply set the bone id to use.
The closest analogue is Geodesic Voxel Heat Diffuse Skinning, which is similar, and seems magical, but is actually still tedious since it is being applied to an existing bone rig that still has to be setup in a tedious way. This workflow is (and has always been) backwards to me since the “bone” concept still a user-facing construct that has no bearing on what is actually happening behind the scenes, and as such, simply “painting” the verts that should belong to each bone with a different color, then painting in a blend value from black-to-white (using the vertex alpha channel), seems much more straightforward to me.
Besides, allowing one to generate both the bones and the rig at the same time would save loads of time when you’ve got a character you might prefer to be more “manual” with yet still don’t know exactly how you need your rig to be setup (and would like to try out and tweak skinning in-engine until you’ve got the blending correct). Being able to quickly regenerate the weights (and the rig) using nothing but colors, point cloud volumes, and point/distance relationships to determine the bones, and then an outliner that shows bone-chain hierarchy, letting you reorder and apply hierarchical relationships to a multi-selected set of bones quickly (based on the order they were selected), would be an amazingly useful rigging tool. I hope you get time to play around with this! – Something like this sounds very promising! 
