How to create a 2D sprite with a 3d skeleton?

[pertholdth]

I have been watching the dev of this game and they have mapped billboarded sprites on a 3d skeleton

here is the link to their system

I was wondering if this could be done in unity

obviously, this is no small feat and there are many moving parts, so I figured I will try to explore it one step at a time

For a start, my question is:

How can I create bones to form a skeleton in unity?

From what I gather, .fbx files are usually modeled, rigged and animated in 3d software like blender, and only after that they are imported into unity, where you can only blend the animations

I was wondering if there was an out of the box feature that would let me start creating bones and mapping textures to the bones, and I would tentavely start from there

If thats not the way you would tackle this problem, your alternative advice would be much appreciated

I understand this is the kind of “if you have to ask about it, you arent equipped to do it” - kind of question, but I would still like to give it a try just to set myself at ease before discarding the idea, I wanted to at least give it a try

[Kurt-Dekker]

@pertholdth

How can I create bones to form a skeleton in unity?

You can actually do it by hand right in Unity, just by creating a hierarchy of invisible GameObjects in whatever armature / skeleton shape you want.

Then you target your animations to those bones and after that, you can “hang” any kinds of geometry you want on the different bones, including sprites if you want. Those things move because the underlying bone moves, not because the animator drives the newly-added thing.

Then you can set those sprites to billboard and change out depending on direction to camera, like the head in the above linked Youtube.

@pertholdth

this is no small feat and there are many moving parts

Oh amen for sure. Start small… think like a pair of chopsticks with a rubber band at the top walking along… that’s doable in 15 minutes or so once you’re familiar with things… then play with knees, body, and gradually get more comfortable at a full skeleton, etc.

There is an IK package out there too which can make doing walk cycles a lot simpler.

EDIT: just to remind myself, I quick-cheesed a little chopstick leg dude… see attached project, made 100% in Unity except for the lowres textures I photobashed off the google image search.

Take your favorite head sprite, remove the ball / eyes in my project, and hang your head sprite there. Then swap it out depending on heading to camera, like front head, side head, quarter-side head, back of head, etc.

9877608–1424481–AnimatedDude.unitypackage (198 KB)

[seanrich92]

It is possible to create this kind of system but it would take a lot of trial and error on your part. While I wouldn’t say I created the same exact system that they have, I made a version that suits my needs based on what they have publicly shown on their tweets. You just need to tweak whatever you need for your own game. Vector math, shader programming and custom editor experience are things that you might need here.

You can see what my system can do which was based on on Project Terra’s ideas

Bone Sprite Animation System

[pertholdth]

@seanrich92

It is possible to create this kind of system but it would take a lot of trial and error on your part. While I wouldn’t say I created the same exact system that they have, I made a version that suits my needs based on what they have publicly shown on their tweets. You just need to tweak whatever you need for your own game. Vector math, shader programming and custom editor experience are things that you might need here.

You can see what my system can do which was based on on Project Terra’s ideas
https://seanrich92.github.io/tools/bone-sprite-animation-system

in your estimation, lets say you wanted to repurpose your system to animate a horse

would it be a whole new thing since the horse has a completely different anatomy

or do you think that all the work you laid would make it easier?

overall how much time would it take you to repurpose it to a horse, since the main system took you 5 months?

thanks for the advice!

[seanrich92]

Probably just setup the bones through my editor, slap in the sprites through another editor tool then adjust the direction and pivot there. Then theoretically it should work already. The creation of the sprites would probably take the most time. It is important to augment your workflow with a lot of custom editors.

[pertholdth]

@seanrich92

Probably just setup the bones through my editor, slap in the sprites through another editor tool then adjust the direction and pivot there. Then theoretically it should work already. The creation of the sprites would probably take the most time. It is important to augment your workflow with a lot of custom editors.

and what does your output sprite look like in 3d space?

I assume that your game is 2d so you “flatten” out the final result? And it would be as if it was a normally drawn sprite (flat as in, it has no z value in an orthographic view)

Or does the final result have some 3 dimensionality?

[seanrich92]

Sprites are always “flat” but you can add some dimensionality through shaders by rotating them in a dimension. You might want to do this only on certain body parts that should look skewed depending on the angle towards the camera.
I do this by changing an enum on my scriptable object that contains the sprite data which adds some value in the shader to do this. Some sprite data do not skew with respect to the camera while some do. It is really up to you which makes sense and which doesn’t. You can see an example of this from the sword gif in my blog.

[pertholdth]

@seanrich92

Sprites are always “flat” but you can add some dimensionality through shaders by rotating them in a dimension. You might want to do this only on certain body parts that should look skewed depending on the angle towards the camera.
I do this by changing an enum on my scriptable object that contains the sprite data which adds some value in the shader to do this. Some sprite data do not skew with respect to the camera while some do. It is really up to you which makes sense and which doesn’t. You can see an example of this from the sword gif in my blog.

thanks

I was referring to the whole composition on the skeleton, for example, there is a torso sized distance inbetween the right arm and the left arm, if you change the perspective to a side view, the right arm is now supposed to be closer to the camera and the left arm further away

this is what I mean by sprites existing in a 3D space

correct me if im wrong, I assume that you finish by displaying the whole combination of small sprites as 1 single object and like the rest of your game, it doesnt use the z axis

I ask because in project terra they are using 3d terrain and Im not sure if they also allow their characters body parts to exist in 3D to some extent, here is one example of what I mean

on this timestamp

if you see when the char falls on the water, the foam that displays on the intersection between the water plane and the feet is not being display at the same height on the sprite

and finally here, the whole sword on a frame that is on the lower section of the sprite, should be fully underwater if a water plane was intersecting normally with a fully flat sprite

however only the feet are shown underwater and the sword is displayed above the water

this could be done with just masking, its true, but it could also just be that there is actually some tridimensionality and the sword does not intersect with the water plane due to being on a greater height value

any way it seems there is some segmenting of body parts and how they are treated differently when intersecting with the water, a character slashing a sword southward would result in the sword showing fully submerged since there would be no way to distinguish relative height between the sword arm and the feet

as you see, if nothing is done, the sword will display underwater

I hope you understand what I mean

please let me know your thoughts, appreciate the advice once again

[seanrich92]

How sprites interact with your terrain/water mesh is an implementation that you have to think about as I didn’t need to do anything like that. The problem is that with unity you have less control over components like sprites over a bespoke engine as sprites, meshes etc are part of unity internal engine code. You might need to just use meshes instead of sprites to have them work better and use shaders to get the behavior that you want.

[pertholdth]

@Kurt-Dekker

You can actually do it by hand right in Unity, just by creating a hierarchy of invisible GameObjects in whatever armature / skeleton shape you want.

Then you target your animations to those bones and after that, you can “hang” any kinds of geometry you want on the different bones, including sprites if you want. Those things move because the underlying bone moves, not because the animator drives the newly-added thing.

Then you can set those sprites to billboard and change out depending on direction to camera, like the head in the above linked Youtube.

Oh amen for sure. Start small… think like a pair of chopsticks with a rubber band at the top walking along… that’s doable in 15 minutes or so once you’re familiar with things… then play with knees, body, and gradually get more comfortable at a full skeleton, etc.

There is an IK package out there too which can make doing walk cycles a lot simpler.

EDIT: just to remind myself, I quick-cheesed a little chopstick leg dude… see attached project, made 100% in Unity except for the lowres textures I photobashed off the google image search.

Take your favorite head sprite, remove the ball / eyes in my project, and hang your head sprite there. Then swap it out depending on heading to camera, like front head, side head, quarter-side head, back of head, etc.

could you tell me what is the proper way to create a rig? From what I gather there are joints and bones? bones are like represented by diamonds and joints connect the diamonds. You can stretch and rotate ( the joints or the diamonds)

I could spitball and do something that somehow works but I feel like its important to get this behaviour right, so that im not reinventing the wheel

(is there a specific reason bones are represented as diamonds)