Topic by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Topic: Mobile-friendly web app for remote controlling Pianoteq 7.5

Hi all! After the Pianoteq 7.5 update, I've developed a very small proof-of-concept, mobile-friendly web app that can remotely control a Pianoteq instance. Right now, it can only control the presets and the output mode (which is what I mostly need it for) and it looks like the following:

https://github.com/shuhaowu/remote-for-pianoteq/raw/master/remote-for-pianoteq-small.jpg

This is released as open source here: https://github.com/shuhaowu/remote-for-pianoteq. Instructions on running this are on that repository. Things are a bit rough for now, but this provides a good base to build more upon. That said, I don't want to have too much bloat on this application as it's meant to run on a low-end phone.

The way I'm using it is as follows:

  • Pianoteq runs on a Raspberry Pi 4 with a PREEMPT_RT kernel on boot. With this, I can achieve pop-free real-time playing with 22kHz internal sampling rate, 96 polyphony, and a buffer size of 128 via ALSA. If there's interest, I can separately release a tutorial or even a flashable image for this.

  • The server.py from this project runs on the Raspberry Pi 4 on boot.

  • I have an older, low-end Android phone that is stored near my piano. I use this device to control Pianoteq via this interface. In a pinch, I can also use this phone to VNC into the Raspberry Pi to change some things not available over the API.

There are some known issues:

  • I can't seem to load my own custom presets without it saying "preset load failed". This seems to be a problem on Modartt's side? Alternatively I could be using the API wrong (doubt it, because I can switch between the built-in presets just fine).

  • If an error occurred while changing settings, the UI will be stuck in a disabled state. This is because the error handling code hasn't been written yet as this is still a proof-of-concept. I will fix this at some point. Pull request is also welcome.

There are some other features that I want, but I can't implement the because the JSON-RPC API doesn't appear to be "complete":

  • The ability to control reverb presets. With the API available in 7.5, you can adjust individual reverb settings but not presets (at least as far as I can see).

  • Start and stop the metronome as well as setting the tempo. Also not supported by 7.5 API as far as I can tell.

  • Ability to control the velocity curve preset. Again, not available in the API as far as I can tell.

Last edited by shuhao (27-12-2021 01:49)

Reply by dv

  • Avatar
  • dv
  • Member
  • Offline
  • Registered: 01-11-2021
  • Posts: 303

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

shuhao wrote:

This is released as open source here: https://github.com/shuhaowu/remote-for-pianoteq. Instructions on running this are on that repository.

Awesome, thanks!!

shuhao wrote:

Pianoteq runs on a Raspberry Pi 4 with a PREEMPT_RT kernel on boot. With this, I can achieve pop-free real-time playing with 22kHz internal sampling rate, 96

There probably are other instructions similar to yours, but if it does not take you too long, it would be very nice to have everything in the same repo! So yes, please. I am indeed planning to have a similar setup in a couple of months, and your instructions would help me tremendously!

I am quite busy with other projects at the moment, but I starred and watched your repo and hope to contribute one day.

Where do I find a list of all posts I upvoted? :(

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

dv wrote:
shuhao wrote:

Pianoteq runs on a Raspberry Pi 4 with a PREEMPT_RT kernel on boot. With this, I can achieve pop-free real-time playing with 22kHz internal sampling rate, 96

There probably are other instructions similar to yours, but if it does not take you too long, it would be very nice to have everything in the same repo! So yes, please. I am indeed planning to have a similar setup in a couple of months, and your instructions would help me tremendously!.

I didn't find any other tutorials with PREEMPT_RT. I found that the RT kernel made a significant difference in latency. This makes a lot of sense as I've observed worst case jitters of ~3-5ms with the stock kernel (as tested via cyclictest), and possibly even more when some processing tasks occur (although that's observed on my x86 desktop, but should be the same principle). If you do the math, the time remaining to fill the buffer before the deadline decreases drastically. With PREEMPT_RT, I've observed a max scheduling latency of something like 200us. Pianoteq appears to correctly set it own priority, so with the PREEMPT_RT kernel, the overrun potential is much less. I don't 100% have proof that this is the reason of the buffer overflow in a non-RT kernel yet. I'm hoping that at some point I can use ftrace to definitively trace this.

It's a bit tricky to setup PREEMPT_RT properly at the moment. It won't likely be in that repo, but I will definitely link to the RT setup repo when I get a chance to work on it.

Reply by mprimrose

  • Avatar
  • mprimrose
  • Member
  • Offline
  • From: Otautahi, Aotearoa
  • Registered: 08-02-2021
  • Posts: 76

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

shuhoa,

I have been having a play with this proof of concept web app on my Debian 9 system and it seems to be running fine. I can change between the built-in presets with no difficulty and the preset on the Pianoteq app changes as requested. However, I run into the same problem as you do i.e.

"I can't seem to load my own custom presets without it saying "preset load failed". This seems to be a problem on Modartt's side? Alternatively I could be using the API wrong (doubt it, because I can switch between the built-in presets just fine)."

My morphed and layered presets show up in the drop down list of presets, but if I try to load them the Web app freezes. Interestingly none of my other presets show up in the drop down list unless I "load an FXP file" in the Pianoteq app before I start the web app, in which case that, and only that preset, shows up in the drop down list. If I change away from it the preset then vanishes from my list.

Which leads me to ask the naive question

"How does the Web app know what the file path to MyPresets actually is?"

Obviously the Pianoteq app knows that the morph and layers are in ~.local/share/Mordartt/Pianoteq/ Presets/My\ Presets (the default) as this is set in Preferences (even though it has problems loading them). However, the FXP presets that I have down loaded from the Mordartt FXP Corner are sitting in ~Mordartt/MyPresets which the Pianoteq app knows about but I don't think the Web app does.

I am wondering whether you need to include the ability to specify the path to a MyPresets folder utilising something like

"loadFile(path)
Load any supported file (fxp/mfxp/scl/kbm/ptq/wav.."

from the API? You could define the paths in a user configuration file or have a data field on the Web app that the user could input the path into. In my ~.config/Modartt/Pianoteq75.prefs file there is a reference to the path to the FXP downloads in

"<VALUE name="RecentFXP" val="~Mordartt/MyPresets/<filename>>"

which you may want the Web app to reference.

I'm not sure it resolves the problem of why the Web app can see, but still not load my morphed and layered presets, but it may explain why the Web app can't see my downloaded presets.

All that to one side, this is a really neat bit of work. Congratulations!!

Michael

Pianoteq 8 Studio plus all Instrument packs; Organteq 2; Debian; Reaper; Carla

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

I haven't tried FXP either. I suspect the reason that it doesn't work will be similar to the custom presets, and appears to be an issue with Pianoteq. Hopefully there's a patch soon (or someone can tell me I'm using it wrong...).

I won't be able to fix the UI freeze on error issue (this is a known problem) and other issues for the next few days, but pull requests are welcome and I can merge stuff if anyone manages to find a fix before I get to it.

Reply by julien

  • Avatar
  • julien
  • Admin
  • Offline
  • Registered: 19-04-2007
  • Posts: 1,009

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

shuhao wrote:

There are some known issues:

  • I can't seem to load my own custom presets without it saying "preset load failed". This seems to be a problem on Modartt's side? Alternatively I could be using the API wrong (doubt it, because I can switch between the built-in presets just fine).

  • If an error occurred while changing settings, the UI will be stuck in a disabled state. This is because the error handling code hasn't been written yet as this is still a proof-of-concept. I will fix this at some point. Pull request is also welcome.

There are some other features that I want, but I can't implement the because the JSON-RPC API doesn't appear to be "complete":

  • The ability to control reverb presets. With the API available in 7.5, you can adjust individual reverb settings but not presets (at least as far as I can see).

  • Start and stop the metronome as well as setting the tempo. Also not supported by 7.5 API as far as I can tell.

  • Ability to control the velocity curve preset. Again, not available in the API as far as I can tell.

Hi shuhao,

That's a nice tool !

You should be able to load custom presets, but you have to provide the 'bank' name. For example, in your code, I believe the syntax would be 

rpc("loadPreset", {"name": name, "bank": "My Presets"})

.  Empty string in the bank value is for loading built-in presets.

And if you want to load a reverb preset, you also have to supply the correct  preset type:

rpc("loadPreset", {"name": name, "preset_type": "reverb"})

For velocity presets, the preset type is "vel" (see the loadPreset documentation).

Regarding the metronome, you are  correct, it is not yet available in the interface.

julien's Website

Reply by navindra

  • Avatar
  • navindra
  • Member
  • Offline
  • Registered: 06-08-2020
  • Posts: 115

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

Bravo! This is exactly what I wanted as well. Perfect start.

Does the MIDI sequencer still work in headless mode with the JSON-RPC API? This is the only thing that would be a blocker for me since I would still have to launch Pianoteq in graphical mode and waste those cycles in addition to now hosting a web server.

Find me at the new piano forums @ PianoTell.com.

Bluebird by Alexis Ffrench on Pianoteq NY Steinway D.

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

julien wrote:

You should be able to load custom presets, but you have to provide the 'bank' name. For example, in your code, I believe the syntax would be 

rpc("loadPreset", {"name": name, "bank": "My Presets"})

.  Empty string in the bank value is for loading built-in presets.

And if you want to load a reverb preset, you also have to supply the correct  preset type:

rpc("loadPreset", {"name": name, "preset_type": "reverb"})

For velocity presets, the preset type is "vel" (see the loadPreset documentation).

Regarding the metronome, you are  correct, it is not yet available in the interface.

Thanks a lot for that info! I'll try to add it quickly into the code. I should have read the docs a bit more carefully as I didn't fully get it when I first read it.

navindra wrote:

Does the MIDI sequencer still work in headless mode with the JSON-RPC API? This is the only thing that would be a blocker for me since I would still have to launch Pianoteq in graphical mode and waste those cycles in addition to now hosting a web server.

Are you talking about the ability to play and record midi sequences? Those are available in the API but I don't really know how to create an UI for it as I haven't really used it. My code should be relatively straight forward to modify and we might be able to get it merged upstream.

As far as CPU cycles hosting a web server: I don't think it should be that big of a deal, especially on Linux. When you're not doing anything it's idling. When you're changing settings, you'll hear some stutter anyway as Pianoteq loads the new settings.

Reply by navindra

  • Avatar
  • navindra
  • Member
  • Offline
  • Registered: 06-08-2020
  • Posts: 115

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

shuhao wrote:

Are you talking about the ability to play and record midi sequences? Those are available in the API but I don't really know how to create an UI for it as I haven't really used it. My code should be relatively straight forward to modify and we might be able to get it merged upstream.

As far as CPU cycles hosting a web server: I don't think it should be that big of a deal, especially on Linux. When you're not doing anything it's idling. When you're changing settings, you'll hear some stutter anyway as Pianoteq loads the new settings.

Yes, this is what I meant. Before JSON-RPC, I could control Pianoteq remotely via MIDI. However, the ability to play and record MIDI sequences doesn't work in headless mode, so I still have to run Pianoteq in full graphical mode anyway.

It's the CPU cycles of Pianoteq drawing and rendering its interface that I want to save by using JSON-RPC and Pianoteq in headless mode. Hopefully that's possible.

Last edited by navindra (30-12-2021 00:55)
Find me at the new piano forums @ PianoTell.com.

Bluebird by Alexis Ffrench on Pianoteq NY Steinway D.

Reply by julien

  • Avatar
  • julien
  • Admin
  • Offline
  • Registered: 19-04-2007
  • Posts: 1,009

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

navindra wrote:

However, the ability to play and record MIDI sequences doesn't work in headless mode, so I still have to run Pianoteq in full graphical mode anyway.

You are correct, this will be fixed in the next update.

julien's Website

Reply by navindra

  • Avatar
  • navindra
  • Member
  • Offline
  • Registered: 06-08-2020
  • Posts: 115

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

julien wrote:
navindra wrote:

However, the ability to play and record MIDI sequences doesn't work in headless mode, so I still have to run Pianoteq in full graphical mode anyway.

You are correct, this will be fixed in the next update.

Wonderful! Thank you, Julien! I thought it was by design.

Find me at the new piano forums @ PianoTell.com.

Bluebird by Alexis Ffrench on Pianoteq NY Steinway D.

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

navindra wrote:

It's the CPU cycles of Pianoteq drawing and rendering its interface that I want to save by using JSON-RPC and Pianoteq in headless mode. Hopefully that's possible.

As a note: I just fully implemented Pianoteq on a PREEMPT_RT (realtime patch) enabled Linux instance on the Raspberry Pi. Pianoteq is running in full graphical mode on a standalone (not scraping) VNC server. Even with a VNC client connected, it's not really impacting the performance of Pianoteq (96 polyphony, 22khz internal sampling rate, 128 buffer). This is because the PREEMPT_RT scheduler is correctly scheduling Pianoteq to process everything on time. Thus I don't think the CPU cycles of interface drawing is that significant, as long as RT scheduling is configured correctly. At some point I might do some tracing to "prove" this, if I get the time to optimize this setup...

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

I can't seem to edit my original post anymore to update the known issues list, but you can now change to a custom preset with this app. You can also change to all of the builtin reverb presets with a new drop down. Thanks again to Julien for pointing me to the right direction.

However, there are a few small issues:

1. I still don't know how to load a FXP. It doesn't seem like FXP is a preset returned by the getListOfPresets API call.
2. When changing to a custom preset, its name in the UI will be "My Presets/<PRESET NAME>". "<PRESET NAME>" will also be in the drop down, due to how the code is written right now to be as simple as possible. Technically, the app just displays whatever is returned by the getInfo API call (which has the "My Presets" part in it). This is not a big deal, so I'm not fixing it for now.
3. You can only change to built-in reverbs. I can't figure out how to enumerate a list of available reverb presets like I can via the getListOfPresets API call.
4. Some built-in piano presets don't seem to use the standard reverb presets (technically: current_preset.mini_presets.reverb == undefined in the getInfo call). In this case, the UI displays "Unknown/Custom". Again, not really an issue, but something worth pointing out.
5. I can't easily change the velocity curve presets because there seems to be no way to enumerate a list of available velocity curve presets.

In any case, this works. The screenshot in the original post has been updated, as it linked to Github.

Reply by navindra

  • Avatar
  • navindra
  • Member
  • Offline
  • Registered: 06-08-2020
  • Posts: 115

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

shuhao wrote:
navindra wrote:

It's the CPU cycles of Pianoteq drawing and rendering its interface that I want to save by using JSON-RPC and Pianoteq in headless mode. Hopefully that's possible.

As a note: I just fully implemented Pianoteq on a PREEMPT_RT (realtime patch) enabled Linux instance on the Raspberry Pi. Pianoteq is running in full graphical mode on a standalone (not scraping) VNC server. Even with a VNC client connected, it's not really impacting the performance of Pianoteq (96 polyphony, 22khz internal sampling rate, 128 buffer). This is because the PREEMPT_RT scheduler is correctly scheduling Pianoteq to process everything on time. Thus I don't think the CPU cycles of interface drawing is that significant, as long as RT scheduling is configured correctly. At some point I might do some tracing to "prove" this, if I get the time to optimize this setup...

I also run Pianoteq on a standalone VNC server but with stock kernel. The key is using the 64-bit OS image, overclocking, and applying all of Modartt's recommendations. My external USB interface may be a factor as well, I'm unsure.

I removed a bunch of unneeded things to get cold boot under 20 seconds.

I seem to be getting better performance than you:

https://forum.modartt.com/viewtopic.php...30#p973230

Sample rate: 44100 Hz (I was actually using it at 48 KHz for the longest time)
Audio Buffer size: 64 samples (1.5 ms)

I have maximum polyphony at 256 but it will clearly have problems if I get anywhere near that. I'm sure I can't get to 96 either, but in my normal playing I never hit this. In the stress test I did at the above link, I hit 48 or more polyphony. It's really hard to hit this in normal playing.

Things get more tough when you use more mics -- this is where things break down for real time playing. Fortunately, I don't need many mics for a player perspective and generating the final recording does not need to be in real time.

Last edited by navindra (31-12-2021 08:01)
Find me at the new piano forums @ PianoTell.com.

Bluebird by Alexis Ffrench on Pianoteq NY Steinway D.

Reply by shuhao

  • Avatar
  • shuhao
  • Member
  • Offline
  • Registered: 24-12-2021
  • Posts: 8

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

navindra wrote:

I also run Pianoteq on a standalone VNC server but with stock kernel. The key is using the 64-bit OS image, overclocking, and applying all of Modartt's recommendations. My external USB interface may be a factor as well, I'm unsure.

I removed a bunch of unneeded things to get cold boot under 20 seconds.

I seem to be getting better performance than you:

https://forum.modartt.com/viewtopic.php...30#p973230

Sample rate: 44100 Hz (I was actually using it at 48 KHz for the longest time)
Audio Buffer size: 64 samples (1.5 ms)

I have maximum polyphony at 256 but it will clearly have problems if I get anywhere near that. I'm sure I can't get to 96 either, but in my normal playing I never hit this. In the stress test I did at the above link, I hit 48 or more polyphony. It's really hard to hit this in normal playing.

Things get more tough when you use more mics -- this is where things break down for real time playing. Fortunately, I don't need many mics for a player perspective and generating the final recording does not need to be in real time.

As a note, the RT kernel is more about consistent performance rather than getting more performance to support higher polyphony. In other words, with the RT kernel I should be able to have no audio dropouts while supporting the same amount of polyphony. I'm not sure if you notice audio drops or cracks when you're playing (especially at higher polyphony), but based on my previous experience with the non-RT kernel and the Raspberry Pi (or any Linux computer), there will be periodic audio dropouts due to scheduling latency.

That said, I've looked through your post and it seems like we're getting relatively similar performance. I noted that the audio load plot that is shown here is the same as what I get when I use buffer size = 64 at 44kHz with a polyphony of 48. There will never be any drop out if I set the max polyphony to 48 with RT. However, at around the 50-60 polyphony mark, the processor becomes overloaded and cannot keep up, causing drop out. Setting a max polyphony higher than what you use doesn't mean the polyphony being rendered by Pianoteq is higher, which is why I can make the comparison between what you observe and what I observe.

With a RT kernel (and a 2GHz overclock), you can get to 96 polyphony at 22khz internal sampling and 128 buffer size without any dropouts indefinitely (unless there are something else overloading the system). You can keep polyphony at 96 (or even more) if you hold down the sustain pedal and glide your fingers continuously up and down the keyboard. The audio load will be about 70% at polyphony ~= 96, as shown by this:

http://files.shuhao.fastmail.com.user.fm/images/Screenshot%20from%202021-12-31%2010-28-46.png

If you already have Linux setup, you can simply install the a Linux RT kernel to get this type of consistent behavior without any audio drop outs. I'm using the RT kernel builder with a patch I made to it here. I'm hoping that I can produce a flashable image for the Raspberry Pi with everything preconfigured soon (including this web app and the VNC server).

Last edited by shuhao (31-12-2021 16:52)

Reply by MartinGr

  • Avatar
  • MartinGr
  • Member
  • Offline
  • Registered: 04-02-2020
  • Posts: 53

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

I'm running Pianoteq 7.5 on an Intel NUC (Windows 11).
I've installed Python 3.10 and call "server.py" as described.
But when I try to open the remote in a browser, it says "Error: failed to perform JSON RPC: 502 Bad Gateway".
Do you have an idea what could be the problem?

Regards,
Martin

Reply by creaktive

  • Avatar
  • creaktive
  • Member
  • Offline
  • From: Amsterdam
  • Registered: 13-07-2022
  • Posts: 6

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

MartinGr wrote:

I'm running Pianoteq 7.5 on an Intel NUC (Windows 11).
I've installed Python 3.10 and call "server.py" as described.
But when I try to open the remote in a browser, it says "Error: failed to perform JSON RPC: 502 Bad Gateway".
Do you have an idea what could be the problem?

Regards,
Martin

Is PianoTeq running with the --serve command line option? By default the JSON RPC interface is deactivated.

Edit: as a follow-up, can you access the interface directly, without server.py running? Try to go to http://127.0.0.1:8081/jsonrpc and see if it loads.

Last edited by creaktive (20-07-2022 07:30)

Reply by MartinGr

  • Avatar
  • MartinGr
  • Member
  • Offline
  • Registered: 04-02-2020
  • Posts: 53

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

creaktive wrote:

Is PianoTeq running with the --serve command line option? By default the JSON RPC interface is deactivated.

Edit: as a follow-up, can you access the interface directly, without server.py running? Try to go to http://127.0.0.1:8081/jsonrpc and see if it loads.

I've checked it - and it didn't load. But it was my fault. I forgot to include "" after --serve

Thanks a lot for this useful tool!

Reply by dubc

  • Avatar
  • dubc
  • Member
  • Offline
  • Registered: 07-08-2022
  • Posts: 22

Re: Mobile-friendly web app for remote controlling Pianoteq 7.5

Great little project you have here! +1 for python!

There is also https://www.touch-portal.com/ which may be useful if you just need single clicks to toggle on / off.