For my master thesis (computer science/software engineering/math) I am making an avatar creator that should be able to make an avatar for a player from a selfie or two.
The approach is to use ASM (Active Shape Models, http://www.ee.oulu.fi/research/imag/courses/Kokkinos/asm_overview.pdf) to find the place and shape of different parts of the face (nose, eyes etc) edit an UMA character face model based on those shapes and then edit the texture from image(s).
Maybe if this works well it will end up on the asset store in the future
Below you can see current result from a single selfie image, result will hopefully improve as I add the possibility to take a photo from the side as well.
Here’s a quick sitrep. Texturing from two views work but it requires quite a bit of manual work right now. Also there is now automatic shape editing of the UMA character face yet but that is coming!
Ahh the polygons visible in the image are only for image segmentation and image-to-texture mapping, they do not make up the model itself. It is possible to define new shapes in the project if more polygons are needed for correct mapping though.
Here’s a quick example where I created an UMA character from two photos and then let RAIN control the character while patrolling around a scene. I did do some post procesing of the textures here as I have yet to find a good solution for filling the whole texture when there is lacking image data. I need to figure out how to do a “hairmask” and a “skinmask” and then do procedural texturing of the parts of the texture that have not been filled.
One more example of an IRL-person turned UMA avatar. We’re getting there! Right now I am working on using statistical analysis (namely finding coordinates of shapes in eigenspaces using PCA Principal component analysis - Wikipedia ) to determine the UMA DNA for the face based on the photos that are segmented. It’s not a trivial problem and it seems all solutions have some drawback or another.
I have been thinking about other approaches as well, maybe using a purely analytical method reducing some type of cost-function but will try the PCA-route first.
If you have experience with anything similar or have ideas or input I’d love to hear from you! Write below or send me a pm!
That’s great! Only piece of advice I have is that it might be easier to customize the UMA DNAConverterBehavior to match the vectors of your analysis rather than trying to do some two step mapping. They are pretty straightforward.
Thanks that’s some solid advice right there! I had not thought to dive deeper into UMA (not knowing it very well) but I have now had a look at the DNAConverters and I think you are absolutely right. Right now the UMA sliders do not correlate very well with my eigenspace coordinates leading to, just as you mention, two step swapping. As principal component analysis by definition finds the most “important” shape differences of the human face I guess it makes sense setting up a DNAConverter that correlates more closely with the eigenvectors from the PCA.
As my database of faces unfortunately is not ethnically diverse enough (consists mostly of male italians… long story…) I doubt that I will find the eigenvectors that best describe human faces but it might be a start
So I was intrigued by the faces of the walking dead game and decided to try and transform my own face to be similar to that. What I did was to use the face generated above that I used in a small demo project and simply draw by hand on top of the texture and then use the handdrawn version instead. Here’s the result, compare to the walking dead-example. Note that I am DEFINITELY no artist, I have no ability at all and still it looks pretty good I’m going to let a friend with more artistic skill have a go at it and try to generate a few handdrawn examples from generated facial textures.
Here are a few more example images of completed characters.
I am currently working on the automated settings of UMA DNA values based on coordinates in the eigenspace spanned by the principal mode eigenvectors. Hitting some snags here and there, I think this may be in part due to a too small face database, but slowly getting there.
Here’s an example of me using the system to generate myself using a webcam. Webcam works but it doesn’t give the most detailed textures and it’s quite hard getting good diffuse light.
The system is pretty much ready for implementing in a game or application as is but a lot refactoring, optimization and user friendliness is needed before it can be released as an asset on asset store.
Putting a shirt on reduces the effect of the “skin transition” that can be seen above. In the future I guess a procedural texture generator forr the skin based on the staatistics of the face skin would be in order but one thing at a time. One thing at a time…
Getting more and more pleased with the results. Not every result is this good, I’d say that this is the best case. Also, textures can’t be better than input photos of course. In this case I used two images taken with a system camera.
If anyone has ideas for and would like to create a demo scene/game/webcomic/movie with this system (or just the avatars, I can also provide the finished avatars to you as a package) please write here or PM me! Of course you will get this package for free in that case!
It would be really cool when presenting my thesis if I could show some stuff others have made using this system.
