Reply by YvesTh

  • Avatar
  • YvesTh
  • Member
  • Offline
  • From: France
  • Registered: 30-06-2020
  • Posts: 213

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Message to the modartt team, if they read it...

For grand pianos the origin point of the head position in binaural mode is on the left at the corner of the keyboard and at floor level, but for upright pianos this point is at the back of the piano. This makes it incompatible to use the creaktive head control program for both pianos. Isn't it ?
Moreover, the apparent length of the grand pianos in the microphone window is the same whatever the real length of the piano, what happens if we position a microphone at the end of the piano? If the microphone is positioned at 2m40 for example, for some pianos it is above the strings, for another it is outside the piano. What does make the software ?

Thank you

Translated with www.DeepL.com/Translator (free version)

Reply by creaktive

  • Avatar
  • creaktive
  • Member
  • Offline
  • From: Amsterdam
  • Registered: 13-07-2022
  • Posts: 6

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

YvesTh wrote:

Hi creaktive,
Thanks again for your work.
Just a question, the head X movement and the head angle seems very close to reality, but the head Y movement is not good for me.
The nose on the webcam or one meter away from the piano give me Y values only from -0.4300 To -0.4400. I can't obtain a position with the head above the keyboard. Perhaps a problem of webcam ? I don't know...

You're welcome, YvesTh!
Interesting; so, it looks like the face detection model does not compute the Y values at all. I did not expect this behavior, but it makes sense: the model recognizes the face mesh but not it's dimensions, so for it (the model) there is no difference between a huge face far away and a tiny one but close to the camera. Fortunately, PianoTeq has the "Head Diameter" parameter which I can use to calibrate for the head size, and infer the actual distance (Y value).
I've just returned from holidays so it will take a while to get around to fixing it...
Regarding your other question: unfortunately I have zero experience with the bluetooth based head trackers (in fact, this thread is the first place where I read about such thing). I was only able to hack together the "face detection + PianoTeq head motion" because, luckily, I already worked on somewhat similar projects

Reply by YvesTh

  • Avatar
  • YvesTh
  • Member
  • Offline
  • From: France
  • Registered: 30-06-2020
  • Posts: 213

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Hi creaktive,
Thank you for taking the time to reply.
Maybe it would be interesting if you could send relative values of head position rather than absolute values. For example I determine a basic position of my head in pianoteq and your software moves the headphone around this point. This would allow to use it for the upright piano. But I don't know if it is possible.

Last edited by YvesTh (02-08-2022 14:17)

Reply by YvesTh

  • Avatar
  • YvesTh
  • Member
  • Offline
  • From: France
  • Registered: 30-06-2020
  • Posts: 213

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Binaural experience, psycoacoustics...

I come back to this post because of a recent experience.

I play mainly with headphones and I really like the "binaural" presets of pianoteq, but even if the sound is interesting I always hear it at the level of my head and I can't imagine the piano in front of me because of the fix head position.
Some time ago I installed on my daw the "wave abbey road studio plugin" with a bluetooth tracker mounted on my headphones. It is supposed to reproduce the feeling of listening to studio monitors. My first feeling in front of my screen was not very convincing, the sound stayed in my head. Then a few days ago, I tried again, headphones on my head I was in front of my keyboard, my computer and 2 monitors that I was not using at that time. I launched pianoteq and the wave plugin, and I played. After a few minutes concentrated on my playing, I suddenly saw one of my monitors and I heard the sound coming out of it, I turned my head and the sound was also coming out of the other monitor. I really thought I had unplugged my headphones and the sound was coming out of the monitors. But no, the sound was coming from the headphones and yet I could hear it a meter in front of me as if it was coming from the speakers, the headphones became transparent. I think that the vision of the monitor triggered a psychoacoustic effect that allowed me to project the sound out of the headphones. Has anyone experienced this sensation ?
It would be great if modartt would develop its "binaural" mode with head tracking to get not the sensation of 2 monitors in front of you but of a real grand piano.
The Creaktive POC above is very interresting but the webcam tracking is not as responsive as nx bluetooth tracking.

Reply by dklein

  • Avatar
  • dklein
  • Member
  • Offline
  • From: Florida
  • Registered: 29-11-2015
  • Posts: 1,495

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Interesting...sounds [sic!] a bit like TrackIR which is widely used in the flight simulator community to adapt visual and audio displays to head position.  I used to use it for flight simulators before VR goggles got good, and I don't know if there is any specific stereo audio driving software for it.

I appreciate the 'magic moment' that you had when you thought that your speakers were playing rather than the headphones.  I remember when I had the first such moment back in late Pianoteq 5 with the Steinway B, thinking that I had Pianoteq muted on my piano's key-reading pick-ups, with the stopbar in, as if I was listening to my acoustic piano, but it was actually PIanoteq with the stopbar blocking the hammers from hitting the strings!

- David

Reply by Qexl

  • Avatar
  • Qexl
  • Member
  • Offline
  • Registered: 04-09-2015
  • Posts: 1,534

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

YvesTh wrote:

installed on my daw the "wave abbey road studio plugin" with a bluetooth tracker mounted on my headphones. It is supposed to reproduce the feeling of listening to studio monitors. My first feeling in front of my screen was not very convincing, the sound stayed in my head. Then a few days ago, I tried again, headphones on my head I was in front of my keyboard, my computer and 2 monitors that I was not using at that time. I launched pianoteq and the wave plugin, and I played. After a few minutes concentrated on my playing, I suddenly saw one of my monitors and I heard the sound coming out of it, I turned my head and the sound was also coming out of the other monitor. I really thought I had unplugged my headphones and the sound was coming out of the monitors. But no, the sound was coming from the headphones and yet I could hear it a meter in front of me as if it was coming from the speakers, the headphones became transparent. I think that the vision of the monitor triggered a psychoacoustic effect that allowed me to project the sound out of the headphones.

YvesTh wrote:

Has anyone experienced this sensation ?


Haha yes - pretty much exactly as you described! More than a few times

It's a good point too, I think maybe getting a visual on my speakers suddenly made me hyper focussed for a time on how real 'the space' was sounding to me.. until I realised the first time it occurred "Oh, but it's pretty late, I'll turn off the speakers and instead wear my headph.. DOH!" the speakers were off, and I was already of course wearing headphones .. er.. happens.. happens ;0)


dklein wrote:

I remember when I had the first such moment back in late Pianoteq 5 with the Steinway B, thinking that I had Pianoteq muted on my piano's key-reading pick-ups, with the stopbar in, as if I was listening to my acoustic piano, but it was actually PIanoteq with the stopbar blocking the hammers from hitting the strings!

Love that David. Pianoteq can certainly seem to be right there.

Pianoteq Studio Bundle (Pro plus all instruments)  - Kawai MP11 digital piano - Yamaha HS8 monitors

Reply by BravoRomeo

  • Avatar
  • BravoRomeo
  • Member
  • Offline
  • Registered: 23-12-2021
  • Posts: 3

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

This is fascinating to me. I use Apple AirPods Pro throughout the day (but not for live piano due to the latency), and one thing that has impressed me is the optional head tracking feature. At one point I forgot I had it enabled and started watching a movie on an airplane. I about panicked as it was a late night flight and the sound seemed to be coming out of my iPad instead of the AirPods. The sensation was so convincing I had to remove the AirPods to check, and then remembered. It’s a neat effect when done well.

The latest version of iOS lets you use the FaceID sensor on an iPhone to scan the shape of your ears and head, then apply that to the binaural model. Based on some interview snippets, I believe the AirPods are computing the binaural head tracking themselves rather than relying on the host device, since Bluetooth latency would be too slow and result in an uneasy feeling. They also position the sound stage relatively, such that wherever you are looking when you hit Play is center forward. If you turn your seat so you are consistently looking a different direction, the algorithm slowly moves center forward to the new direction. They also somehow sync it to the accelerometer in the host device to keep it synchronized if both are moving same way.

Anyway, I’d love to see head track binaural sound added to Pianoteq. I keep dabbling with the demo and will probably procure a license soon. It’s such a fun thing to play with. One of my sheet music apps can use the accelerometers in the AirPods to turn pages, just wag your head. It works quite well. Seems like adding a clip-on accelerometer to a set of wired headphone would be a great way forward.

Reply by Qexl

  • Avatar
  • Qexl
  • Member
  • Offline
  • Registered: 04-09-2015
  • Posts: 1,534

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Wow - thanks for that info BravoRomeo - and welcome.

It'd be wild to have this setup - makes good sense..

BravoRomeo wrote:

I believe the AirPods are computing the binaural head tracking themselves rather than relying on the host device, since Bluetooth latency would be too slow and result in an uneasy feeling

Pianoteq Studio Bundle (Pro plus all instruments)  - Kawai MP11 digital piano - Yamaha HS8 monitors

Reply by BravoRomeo

  • Avatar
  • BravoRomeo
  • Member
  • Offline
  • Registered: 23-12-2021
  • Posts: 3

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

Qexl wrote:

Wow - thanks for that info BravoRomeo - and welcome.

It'd be wild to have this setup - makes good sense..

Sure thing, and thanks!

One way to preview what head-tracked piano might sound like is indeed to use AirPods capable of it connected to a phone or an iPad, then head on over to the Pianoteq demo page and start playing a demo track. While the audio is playing through your AirPods, swipe down from the control center, long press the Volume slider for quick access. Turn on Spatialize Stereo with Head Tracking. It's pretty darn convincing.

Reply by meta-meta

  • Avatar
  • meta-meta
  • Member
  • Offline
  • Registered: 02-04-2021
  • Posts: 4

Re: Binaural, head tracker, virtual reality... why not the PIANOTEQ future

I have a bespoke spatial audio VR playground with virtual instruments, some of which use Pianoteq. I only recently noticed that Pianoteq's binaural mode has head position params. Plugging my head position into these is on my short list of things to setup. As such, I've also been poking around wondering if Pianoteq has OSC (Open Sound Control) support, because I'd quite like to be able to modulate spatially-oriented params like the strike point and pickup distance (MKII) and plucking point (classical guitar) with higher resolution than 0-127 that CC provides.

I have encountered and tried out the JSON-RPC API, which is a blessing for sure! I do not think it is well suited for fast updates however and I am guessing the latency experienced is due to making updates over HTTP requests.

I'll try mapping my head position to MIDI CC (since my app already talks to MIDI devices including a loopMIDI loopback device I've setup for VR instruments). I anticipate that it will be lower latency than JSON-RPC but will suffer from discrete jumps in position because of CC resolution.

Anyone know what the precision is for Pianoteq parameters? Are a lot of them floating point numbers under the hood?

The other limitation that prevents Pianoteq from offering head-tracked binaural simulation is that there is only 1 rotational degree of freedom. I'm sure I can have fun with head tracking, as-is but I think a much better head-tracked experience can be had with OSC messages, or possibly MIDI 2.0. For my VR music playground, I have had a good time using IEM's ambisonic plugins and feel that they are a great example of an OSC API: https://plugins.iem.at/docs/osc/#scenerotator