• Showcase

  • Spine Pro Vtuber Working Prototype

    inglés
Related Discussions
...

Thank you for explaining about them! It seems like a lot of things worked better by using the track time instead of using track alpha to change animations, right? That’s an interesting finding.

The benefit of using the track time is much noticeable on your character model than my test model. Weighted meshes behave better when they are animated on a timeline as opposed to being a single keyframe.

I started using the track alpha because I saw it in a demonstration code and mimicked the usage. I had to research the Spine runtime documentation to find information about track time.

By the way, this change has also made the creation of the model much easier. It used to be difficult to make sure that the eye's mesh was clean when in the middle of the eye-open animation, but now it is easy to check and modify.

Awesome! I always wanted the creation process to be easy.

I think the animation of turning the face sideways could be improved, and I would like to modify it eventually.

Are speaking about your character model or the animation tracks setup in the Spine Vtube Prototype?

Are speaking about your character model or the animation tracks setup in the Spine Vtube Prototype?

Oops, sorry for not being obvious about it. I meant to talk about improvements to my model. My current model has just a narrow angle that can be moved left or right, but thanks to the ability to move the character using the track time, I expect that I should be able to switch the attachment at a specific time and have it face completely sideways. I'm hoping to get that done at some point!

My current model has just a narrow angle that can be moved left or right, but thanks to the ability to move the character using the track time, I expect that I should be able to switch the attachment at a specific time and have it face completely sideways. I'm hoping to get that done at some point!

I am not sure if the face tracking can turn all the way to the side and behave correctly. You can increase the strength of the face yawing signal so you do not have to turn your head completely sideways.

For the next update, I am working on adding index numbers to the face mesh to help sort the confusion with multiple face tracking and corresponding Spine models. I also want to work on saving and loading JSON files for Prototype settings.

I am not sure if the face tracking can turn all the way to the side and behave correctly.

Probably it is no problem, because what I'm planning is that I'd like to make it so that the character moves larger, such as turning sideways about 60 degrees, while the real face only moves at about 40 degrees.

For the next update, I am working on adding index numbers to the face mesh to help sort the confusion with multiple face tracking and corresponding Spine models. I also want to work on saving and loading JSON files for Prototype settings.

Those things would be very useful! I'm looking forward to the next update 😃

I have a working saving and loading mechanism added to the project. The downloadable JSON will have a ( .svp ) extension name to avoid confusing with the Spine exported JSON. I will update https://silverstraw.itch.io/spine-vtuber-prototype in a few days. Hopefully there are not any bugs to fix so I can start working on expanding additional animations and skins implementation.

1.0.6

  • Added a button, located right next to the property drop down menu, to save the model's settings. The model number will determine which settings the button will save from. The downloadable file is formatted as a JSON with ( .svp ) file extension to avoid confusion with Spine's exported JSON file.
  • The drag-and-drop zone and upload file button now reads ( .svp ) file as a JSON text.

https://silverstraw.itch.io/spine-vtuber-prototype

I have updated my skeleton and deleted the old one.
chara-for-Spine-Vtuber-Prototype.zip

@SilverStraw I'm surprised the v1.0.6 release came sooner than I thought! I confirmed that it works fine, but personally I wanted to save the skin setting, so it was a bit sad that the skin is not included in what is saved in the .svp file. (In my case, my skeleton has green and blue eye skins, but it's a bit of a hassle to have to set the skin each time I load it.)

Misaki, I fixed the ( .svp ) file to include the skin with the help of your character model for debugging. I thought about adding skin to ( .svp ) after mix and match skin implementation.

I could have updated sooner but the hot and humid weather delayed me.

Later on the day, after work, I did some experimentation with injecting recorded video into the camera feed. I thought it would be interesting to share. Also, Misaki, I used your model for my test.

My DeviantArt test video because stock video license didn't allow Youtube: https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-55dc604511c0/dfayw2p-8bac9980-e12a-4d89-b384-0978995dbef1.1080p.0ddcac221fef40d48b9618a24a4f0d73.mp4

Stock video from: https://mixkit.co/free-stock-video/closeup-of-young-woman-thinking-about-decision-15776/

Mixkit Restricted License

Misaki, I fixed the ( .svp ) file to include the skin with the help of your character model for debugging. I thought about adding skin to ( .svp ) after mix and match skin implementation.

Great! This is really helpful. Thank you! 😃

I uploaded another video of the current model I updated yesterday with a few more modifications (I made a part of the outline of the face and the side hair disappear when turning face left or right.):

Regarding your new experimentation, it is really interesting! It would be possible to record facial footage outdoors and then replace the face with their Vtuber model based on that video at home. Or simply, it would be easy and helpful to use recorded video for testing. (I always had to call my husband to help me when testing multiple face tracking, so it would be useful if I could record our faces and test multiple face tracking whenever I wanted.)

The heat is really severe, so please be careful of heat stroke!

The heat is really severe, so please be careful of heat stroke!

Yes I am staying cool. Thank you for your concern.

Great! This is really helpful. Thank you! 😃

You are welcome!

Regarding your new experimentation, it is really interesting! It would be possible to record facial footage outdoors and then replace the face with their Vtuber model based on that video at home. Or simply, it would be easy and helpful to use recorded video for testing. (I always had to call my husband to help me when testing multiple face tracking, so it would be useful if I could record our faces and test multiple face tracking whenever I wanted.)

Yes, I plan to include uploading video file for Prototype. I still need more time to work on this feature.

Erika on Discord

[/i][/b][/u]

If only the yes jitter could be fixed, this could be usable! ? maybe with an average slider that the user can self adjust?

Animation Smoothing Function Test. I removed the low-pass jitter filter and replaced it with an unweighted moving average smoothing.

https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-55dc604511c0/dfb3li1-a1190425-4803-495e-943d-7e5e6fdda611.1080p.351b7ec9d4a04555bb1f5979b35b00c7.mp4

  • Nate

  • Publicación n.º 70
  • Publicación en inglés

FWIW, I like this smoothing function (based on ola.js):

function Smooth(f,t){var p=performance,b=p.now(),o=f,s=0,m,n,x,d,k=f;return function(v){n=p.now()-b;m=t[i]t/n/n;d=o-f;x=s[/i]t+d+d;if(n>0)k=n<t?o-x/m/t[i]n+(x+x-d)/m-s[/i]n-d:o;if(v!=void 0&&v!=o){f=k;b+=n;o=v;s=n>0&&n<t?s+3[i]x/m/t-(4[/i]x-d-d)/m/n:0}return k}}

It is very easy to use:

// Setup: initial value, milliseconds over which to smooth
var smooth = new Smooth(0, 350);
// Then later, use the smoothed value returned by this instead of the raw value:
var smoothedValue = smooth(rawValue);

It gives a smoothed value that moves to the raw value over time. It's better than the usual interpolation between a start and end value because it dynamically adjusts the curve smoothly when the raw value is changing a lot. You can see it in action here. Try changing the time from 1000 to 1500 (click Run afterward) for a more interesting demo, showing what happens when the raw value hasn't been reached yet and the raw value changes.

The updates since I last tried are awesome! Very excited about the timeline animation addition for getting inbetweens looking nice. Switching attachments, mesh deformation on eyelids, color changes, all that seems to work great. The skins selector is also nice to have. It does seem smoother too after the smoothing code was added, despite the fact that my lighting is very dim right now and my camera isn't picking up my face too well. :upsidedown: I'm going to make a more complex character rig and see how refined I can get with this. Hopefully I can get that done sometime this weekend. Great work!

I have updated my model, and uploaded a new video using it:

(I noticed after I finished recording that an attachment for the mouth had scaled weirdly and was visible in the front of the back of the hair… This is just my fault.)
The new model just replaced the hair and modified the shape of the face to make it look better, but while I was at it, I did a test of multiple face tracking again. At 0:58 in the video, the index of the face has been replaced, but otherwise it was fine and fun 😃

​1.0.7

  • Removed the low-pass filter for face tracking.
  • Added unweighted moving average smoothing for face tracking.​

https://silverstraw.itch.io/spine-vtuber-prototype

https://itch.io/t/2255285/change-log

FWIW, I like this smoothing function (based on ola.js):

It is very easy to use:

It gives a smoothed value that moves to the raw value over time. It's better than the usual interpolation between a start and end value because it dynamically adjusts the curve smoothly when the raw value is changing a lot. You can see it in action here. Try changing the time from 1000 to 1500 (click Run afterward) for a more interesting demo, showing what happens when the raw value hasn't been reached yet and the raw value changes.

Thank you for the information, Nate.

The updates since I last tried are awesome! Very excited about the timeline animation addition for getting inbetweens looking nice. Switching attachments, mesh deformation on eyelids, color changes, all that seems to work great. The skins selector is also nice to have. It does seem smoother too after the smoothing code was added, despite the fact that my lighting is very dim right now and my camera isn't picking up my face too well. :upsidedown: I'm going to make a more complex character rig and see how refined I can get with this. Hopefully I can get that done sometime this weekend. Great work!

Thank you, Cydoni. Keep me posted with your progress on your character rig. I hope you get some time and not distracted 😉 . Also, congrats on being able to have your wedding ceremony.

(I noticed after I finished recording that an attachment for the mouth had scaled weirdly and was visible in the front of the back of the hair… This is just my fault.)
The new model just replaced the hair and modified the shape of the face to make it look better, but while I was at it, I did a test of multiple face tracking again. At 0:58 in the video, the index of the face has been replaced, but otherwise it was fine and fun 😃

The model came out awesome. The hair is detailed with depth when the head turns. I noticed the mouth part sticking out at the back of the neck 😃 . I don't know if I could fix the face index when the face tracking looses focus on one of the faces. There might be a work around but I haven't figured it out yet. I am still working on trying to integrate uploading a video for face tracking. The prototype 1.0.7 has face tracking smoothing so the models should have a smoother animation. The jitter filter was not working that well.

Can a monkey vtube?

https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/v/mp4/1e351697-997f-420a-b339-55dc604511c0/dfba7l6-b9406265-ba8f-441f-a105-9f8949b3282a.1080p.c7f990c5b4274defbf0bdd3b27c4d2e3.mp4

Wow, ha ha, your tool can even track the monkey's facial movements, awesome! 😃 :grinteeth:
I am glad you are using my new model so soon!

By the way, I was going to try 1.0.7, but I got an error and failed to load the resource:

Failed to load resource: the server responded with a status of 403 ()


Can you think of what might be causing it?

Misaki escribió

Wow, ha ha, your tool can even track the monkey's facial movements, awesome! 😃 :grinteeth:

I tried a cat before but it didn't work too well. Primate facial characteristics are similar to a human so I thought the face tracking AI model might be able to classify a monkey's face.

Misaki escribió

I am glad you are using my new model so soon!

I was researching constraints for future physics constraint. My test model didn't use any constraints but your model does.

Misaki escribió

By the way, I was going to try 1.0.7, but I got an error and failed to load the resource:

That is from my experimentation with loading a recorded video and face tracking for future update. I didn't include the mp4 file in the folder. Prototype should work without breaking.

I tried the 1.0.7 this morning and the skeleton loaded fine. I never tried to upload a video yesterday, just a skeleton file as usual, but perhaps the problem I encountered yesterday might be a server-side problem. Anyway, it certainly seems to be able to move smoother!

I have also made some minor modifications and uploaded the latest model here:
chara-for-Spine-Vtuber-Prototype.zip
The main modification this time was to separate the eye parts into smaller pieces and adjust the weights.

5 días más tarde
Misaki escribió

I never tried to upload a video yesterday, just a skeleton file as usual, but perhaps the problem I encountered yesterday might be a server-side problem.

It was something I forgot remove when I updated the folder on itch.io.

Misaki escribió

Anyway, it certainly seems to be able to move smoother!

It sure does.

Misaki escribió

I have also made some minor modifications and uploaded the latest model here:

Downloaded.

In the next update, the *.svp file will include saving information about the size of the canvas, the background color of the canvas, position and zoom of the viewport. I want to include the ability to use a video for face tracking since OatmealMu on Discord asked about the feature.

https://silverstraw.itch.io/spine-vtuber-prototype

Update 1.0.8
* Allow spaces in Spine animation names to be substituted with hyphens ( - ) or underscores ( _ ).
* Reduce draw face mesh execution time from 5-7 milliseconds to 0-1 milliseconds. Increase frame rate for Spine model while face mesh is shown.
* Expand .svp JSON file to include exporting/importing viewport position, viewport zoom, Spine model canvas height and width, Spine model canvas background color.

  • Add title heading for loading state.
    • Fix permanently disappearing input field after switching between certain model properties .
    • Add viewport zoom property to be editable under 'Single Value Properties'.

@[borrado]
Great, it's very convenient that the viewport and canvas settings are now saved in the .svp file! Also, I am very happy that the improved frame rate allows for quite smooth movement 😃

However, when I look at the console in the developer tools of Chrome, there seems to be some kind of error immediately after opening the page. I am not sure if this has anything to do with it, but sometimes when I try to display the second model, the models turn black. Here is a video of the errors:

I have attached a screenshot of the error log as it may be difficult to see in the video:

This is not a problem when a single model is to be rendered. Could you check the errors?

The HTTP Error 503 is itch.io website issue and should not be caused by Prototype files. lib.js is itch.io file that is added on all web browser games on the website.

I was not able tor reproduce the black shader bug using two models on Chrome. Can you check if it is reoccurring bug? Also, try to load the model files without a .svp file. I wonder if the expansion created some bugs for multiple model loading, but if you did not have an issue with single model then you should not have an issue with multiple models.

The "sample.mp4" is just my testing video that was not included in the Prototype folder. I should be able to include video face tracking in the next update.

Thank you for your reply! I understand each of the error logs.

Can you check if it is reoccurring bug? Also, try to load the model files without a .svp file. I wonder if the expansion created some bugs for multiple model loading, but if you did not have an issue with single model then you should not have an issue with multiple models.

I tried again and confirmed that I can reproduce the bug without the .svp file. Here is the screen recording:

Do you have any idea what might be the cause?