I have encountered and tried out the JSON-RPC API, which is a blessing for sure! I do not think it is well suited for fast updates however and I am guessing the latency experienced is due to making updates over HTTP requests.

I'll try mapping my head position to MIDI CC (since my app already talks to MIDI devices including a loopMIDI loopback device I've setup for VR instruments). I anticipate that it will be lower latency than JSON-RPC but will suffer from discrete jumps in position because of CC resolution.

Anyone know what the precision is for Pianoteq parameters? Are a lot of them floating point numbers under the hood?

The other limitation that prevents Pianoteq from offering head-tracked binaural simulation is that there is only 1 rotational degree of freedom. I'm sure I can have fun with head tracking, as-is but I think a much better head-tracked experience can be had with OSC messages, or possibly MIDI 2.0. For my VR music playground, I have had a good time using IEM's ambisonic plugins and feel that they are a great example of an OSC API: https://plugins.iem.at/docs/osc/#scenerotator

Wow - thanks for that info BravoRomeo - and welcome.

It'd be wild to have this setup - makes good sense..

Sure thing, and thanks!

One way to preview what head-tracked piano might sound like is indeed to use AirPods capable of it connected to a phone or an iPad, then head on over to the Pianoteq demo page and start playing a demo track. While the audio is playing through your AirPods, swipe down from the control center, long press the Volume slider for quick access. Turn on Spatialize Stereo with Head Tracking. It's pretty darn convincing.

It'd be wild to have this setup - makes good sense..

The latest version of iOS lets you use the FaceID sensor on an iPhone to scan the shape of your ears and head, then apply that to the binaural model. Based on some interview snippets, I believe the AirPods are computing the binaural head tracking themselves rather than relying on the host device, since Bluetooth latency would be too slow and result in an uneasy feeling. They also position the sound stage relatively, such that wherever you are looking when you hit Play is center forward. If you turn your seat so you are consistently looking a different direction, the algorithm slowly moves center forward to the new direction. They also somehow sync it to the accelerometer in the host device to keep it synchronized if both are moving same way.

Anyway, I’d love to see head track binaural sound added to Pianoteq. I keep dabbling with the demo and will probably procure a license soon. It’s such a fun thing to play with. One of my sheet music apps can use the accelerometers in the AirPods to turn pages, just wag your head. It works quite well. Seems like adding a clip-on accelerometer to a set of wired headphone would be a great way forward.

installed on my daw the "wave abbey road studio plugin" with a bluetooth tracker mounted on my headphones. It is supposed to reproduce the feeling of listening to studio monitors. My first feeling in front of my screen was not very convincing, the sound stayed in my head. Then a few days ago, I tried again, headphones on my head I was in front of my keyboard, my computer and 2 monitors that I was not using at that time. I launched pianoteq and the wave plugin, and I played. After a few minutes concentrated on my playing, I suddenly saw one of my monitors and I heard the sound coming out of it, I turned my head and the sound was also coming out of the other monitor. I really thought I had unplugged my headphones and the sound was coming out of the monitors. But no, the sound was coming from the headphones and yet I could hear it a meter in front of me as if it was coming from the speakers, the headphones became transparent. I think that the vision of the monitor triggered a psychoacoustic effect that allowed me to project the sound out of the headphones.

Haha yes - pretty much exactly as you described! More than a few times

It's a good point too, I think maybe getting a visual on my speakers suddenly made me hyper focussed for a time on how real 'the space' was sounding to me.. until I realised the first time it occurred "Oh, but it's pretty late, I'll turn off the speakers and instead wear my headph.. DOH!" the speakers were off, and I was already of course wearing headphones .. er.. happens.. happens ;0)

Love that David. Pianoteq can certainly seem to be right there.

I appreciate the 'magic moment' that you had when you thought that your speakers were playing rather than the headphones.  I remember when I had the first such moment back in late Pianoteq 5 with the Steinway B, thinking that I had Pianoteq muted on my piano's key-reading pick-ups, with the stopbar in, as if I was listening to my acoustic piano, but it was actually PIanoteq with the stopbar blocking the hammers from hitting the strings!

I come back to this post because of a recent experience.

I play mainly with headphones and I really like the "binaural" presets of pianoteq, but even if the sound is interesting I always hear it at the level of my head and I can't imagine the piano in front of me because of the fix head position.
Some time ago I installed on my daw the "wave abbey road studio plugin" with a bluetooth tracker mounted on my headphones. It is supposed to reproduce the feeling of listening to studio monitors. My first feeling in front of my screen was not very convincing, the sound stayed in my head. Then a few days ago, I tried again, headphones on my head I was in front of my keyboard, my computer and 2 monitors that I was not using at that time. I launched pianoteq and the wave plugin, and I played. After a few minutes concentrated on my playing, I suddenly saw one of my monitors and I heard the sound coming out of it, I turned my head and the sound was also coming out of the other monitor. I really thought I had unplugged my headphones and the sound was coming out of the monitors. But no, the sound was coming from the headphones and yet I could hear it a meter in front of me as if it was coming from the speakers, the headphones became transparent. I think that the vision of the monitor triggered a psychoacoustic effect that allowed me to project the sound out of the headphones. Has anyone experienced this sensation ?
It would be great if modartt would develop its "binaural" mode with head tracking to get not the sensation of 2 monitors in front of you but of a real grand piano.
The Creaktive POC above is very interresting but the webcam tracking is not as responsive as nx bluetooth tracking.

Hi creaktive,
Thanks again for your work.
Just a question, the head X movement and the head angle seems very close to reality, but the head Y movement is not good for me.
The nose on the webcam or one meter away from the piano give me Y values only from -0.4300 To -0.4400. I can't obtain a position with the head above the keyboard. Perhaps a problem of webcam ? I don't know...

You're welcome, YvesTh!
Interesting; so, it looks like the face detection model does not compute the Y values at all. I did not expect this behavior, but it makes sense: the model recognizes the face mesh but not it's dimensions, so for it (the model) there is no difference between a huge face far away and a tiny one but close to the camera. Fortunately, PianoTeq has the "Head Diameter" parameter which I can use to calibrate for the head size, and infer the actual distance (Y value).
I've just returned from holidays so it will take a while to get around to fixing it...
Regarding your other question: unfortunately I have zero experience with the bluetooth based head trackers (in fact, this thread is the first place where I read about such thing). I was only able to hack together the "face detection + PianoTeq head motion" because, luckily, I already worked on somewhat similar projects

For grand pianos the origin point of the head position in binaural mode is on the left at the corner of the keyboard and at floor level, but for upright pianos this point is at the back of the piano. This makes it incompatible to use the creaktive head control program for both pianos. Isn't it ?
Moreover, the apparent length of the grand pianos in the microphone window is the same whatever the real length of the piano, what happens if we position a microphone at the end of the piano? If the microphone is positioned at 2m40 for example, for some pianos it is above the strings, for another it is outside the piano. What does make the software ?

Thank you

Translated with www.DeepL.com/Translator (free version)

OK I made a PoC using face tracking via webcam: https://sysd.org/pianoteq-pov/
Works in Chrome & Firefox but not Safari (which refuses to call the JSON-RPC interface on localhost due to security concerns).
FWIW, the source code is here.

Hi creaktive,
Thanks again for your work.
Just a question, the head X movement and the head angle seems very close to reality, but the head Y movement is not good for me.
The nose on the webcam or one meter away from the piano give me Y values only from -0.4300 To -0.4400. I can't obtain a position with the head above the keyboard. Perhaps a problem of webcam ? I don't know...

creaktive wrote:

OK I made a PoC using face tracking via webcam: https://sysd.org/pianoteq-pov/
Works in Chrome & Firefox but not Safari (which refuses to call the JSON-RPC interface on localhost due to security concerns).
FWIW, the source code is here.

Hi creaktive,

I've managed to get the face tracking to work on my mobile phone. How do I connect it with Pianoteq on my laptop?

Thank you,

Warmest regards,

Chris

You have to start Pianoteq on the command line, with the option

--serve ""

If you use windows I have no idea how to to that. If you use Linux or Mac you simply type

Pianoteq\ 7 --serve ""

in a shell (possibly adding the full qualified path before Pianoteq).

You can do that before or after opening the browser, it does not matter. If you are running on the same machine, you do nothing else, and it should work. If you are running on a different machine (e.g. phone), you have to find what is the IP address of the machine you are running pianoteq into, and changing into that browser field where it says "localhost" replacing with the IP address of your Pianoteq machine. Also, you must be sure that both the router and machine firewalls allow that communication to happen. It might work out of the box, or it may be a can of worms to debug, but you should be able to find lots of information online in places like stackoverflow.

OK I made a PoC using face tracking via webcam: https://sysd.org/pianoteq-pov/
Works in Chrome & Firefox but not Safari (which refuses to call the JSON-RPC interface on localhost due to security concerns).
FWIW, the source code is here.

Hi créative,
I just came back from holidays and tried your PoC. By clicking on your link I get the head tracking by webcam, very well. But how to make the link with pianoteq? Do I have to open it first? After?  And how? I know absolutely nothing about json, do I have to install something? Thanks for your help...

OK I made a PoC using face tracking via webcam: https://sysd.org/pianoteq-pov/
Works in Chrome & Firefox but not Safari (which refuses to call the JSON-RPC interface on localhost due to security concerns).
FWIW, the source code is here.

Hi creaktive,

I've managed to get the face tracking to work on my mobile phone. How do I connect it with Pianoteq on my laptop?

Thank you,

Warmest regards,

Chris