Topic: Mystery solved!

When you have eliminated the impossible, whatever remains,
however improbable, must be the truth.
Sherlock Holmes

It has been a while since my last post in https://www.forum-pianoteq.com/viewtopic.php?id=6849 .
Somehow this topic is closed so I will post my final post in this new topic, hoping it will serve somebody when encountering the same strange problems as i did.

To summarize the previous topic, I encountered strange latencies.
Sometimes a chord was delayed, sometimes a single note, sometimes I could play for hours without any issues.

After months of trying I could not find the answer so I was ready to sell my license.
But as soon as I posted that my license was for sale, this helpful community reached out with numerous ideas to fix my problems. This made me realize it was worth an other try. You can read the whole story in the post mentioned before.

One important clue was that when running on windows my cpu frequency was at max when selecting the "performance" powersetting. I was running Pianoteq on Linux and my cpu never kept running at max, it was just spiking to max sometimes.
The fix for this problem came from Groovy: cpupower set --perf-bias 0.
Now i had my cpu running at max and Pianoteq was running with real-time priorities, but still the problems keep bugging me.

Tired off trying i decided to go for a quick solution and see how to fix my Pianoteq problems later. I friend of mine was very happy with his Ipad and Ravenscroft piano,
so i decided to try this on a Ipod. The setup on the Ipod was tested extensively in combination with his Nord Stage. I took the Ipod home and connected it to my Numa Compact2. After just a little while i was shocked to hear exactly the same delays that i heard with Pianoteq. Now things got really interesting...
The only thing from my setup that hasn't been replaced is the piano.
But...I replayed the midi files that I recorded with this piano and they were fine. Also, there were no problems to be found in the Pianoteq midi events screen too.

I decided to run a midi monitor(Streambyter) on the Ipod and see if i could find any anomalies.
After a while i encountered my usual delays and started to browse the received midi data. Suddenly something caught my eye:

Ch:01 Note On A1 Vel:081
23:13:45:080

Ch:01 Note On E3 Vel:079
23:13:45:080

Ch:01 Note On A2 Vel:070
23:13:45:080

Whats this? Midi is a sequential protocol (note after note)...
Notes should never arrive at exactly the same time. Even if this was correct,
I'm not a pro... my timing isn't that good... I will never be able to play 3 notes, with 2 hands, at exactly the same time.
But why didn't I saw this with the Pianoteq midi viewer?
It seems that Pianoteq is showing the midi time-stamp, but Streambyter is showing the system-receive time-stamp. Why does this makes a difference?
Well, my recordings played back fine, so the midi time-stamp is correct.
But, when playing live, the timing is wrong. So the events were sent with the right
midi time-stamp, but at the wrong time! This is called midi-jitter.
Fortunately I have a Emu Longboard lying around, so i connected this to the Ipod and started testing... No matter how hard I tried, I could not make it fail.

I confronted Studiologic (Fatar) with this behavior and after a few days i got a reply,
in short: "All our instruments are tested extensively and we never encountered problems like the one you mention, also no other customers complaint about this"
Off course I wasn't satisfied with this answer and after a few mails back and forward  I received a email containing this line:

Please let me know if you find the same jitter with any sound selected, as (for instance) the Organ sounds are based on an additional DSP modeling, very demanding (time wise).

Eureka! My preset that I always used was made of a layer with a pad and a 888 organ. I use a Korg Nanocontrol to select which layer should be active, the pad, the organ, Pianoteq, or a combination of the sounds.
I started playing and as soon as the jitter occurred i changed the organ to a Piano (sampled). Within 10 minutes the jitter was gone. I selected the organ again and low and behold... the jitter came back. I repeated my previous steps and was able to reproduce the results time after time...

So... the Numa Compact has a cpu that is in fact not fitted for the job.
At least if you want to use it to play an organ and use midi at the same time.
For now I removed the organ from my preset and everything seems to run stable.
I tested the modified preset with both Pianoteq and the Ravenscroft piano on the Ipod.

Where does this leaves me?
The Ipod performs very well, so this is a nice addition to my setup.
There are a number of organ apps for Ios so I'll use one of those instead of the
888 from the Numa Compact.

Thanks for all efforts to help me out!

Last edited by MrRoland (03-11-2019 22:03)

Re: Mystery solved!

Man, who knew your keyboard hardware problem, easily, might had been solved without any escalation  —but by a simple, local MIDI off?

Pianoteq 8 Studio Bundle, Pearl malletSTATION EM1, Roland (DRUM SOUND MODULE TD-30, HandSonic 10, AX-1), Akai EWI USB, Yamaha DIGITAL PIANO P-95, M-Audio STUDIOPHILE BX5, Focusrite Saffire PRO 24 DSP.

Re: Mystery solved!

Amen Ptah Ra wrote:

Man, who knew your keyboard hardware problem, easily, might had been solved without any escalation  —by a simple, local MIDI off?

Well, there's the catch...
Local control is off, but the engine is still running because midi receive will always be active. That's what made it so confusing...
I turned local control off, but nothing changed.
Studiologic support finally mentioned that the engine keeps running (and wasting cpu cycles) until you select a sampled instrument.

Re: Mystery solved!

Nice bit of troubleshooting.  And very interesting info on how an apparently independent midi keyboard can throw things out of whack on a midi setup.  I'd never have expected that.  Hope this lets you get back to using Pianoteq (or whatever you choose) to make the sounds you want.

MrRoland wrote:

Studiologic support finally mentioned that the engine keeps running (and wasting cpu cycles) until you select a sampled instrument.

Ah yes, the joys of the "there's no need to shut down that process - it's doing no harm" logic I have heard so many times in my career.  First thing a programmer should be taught - release resources you are not using.

StephenG

Re: Mystery solved!

Thanks for keeping this up-to-date!

If I understand it correctly and try to put it in one sentence:

Your Numa Compact 2 sends MIDI Note-On events sometimes with an unexpected and audible high delay?

Whats this? Midi is a sequential protocol (note after note)...
Notes should never arrive at exactly the same time.

... normally there should be a minimum distance of 0.96 ms, but when your MIDI monitor just has a resolution of 1 ms then some events appear to be "at once" - nothing to worry about.

But why didn't I saw this with the Pianoteq midi viewer?

... accident, because not easy reproducible. Play long enough and you will catch those simultaneous events in PTQ's monitor.

It seems that Pianoteq is showing the midi time-stamp, but Streambyter is showing the system-receive time-stamp. Why does this makes a difference?

... there is no "midi time-stamp" in realtime MIDI events sent from a MIDI keyboard. Your MIDI receiver/app adds a timestamp when a Note-On event is detected. Pianoteq's MIDI event monitor and Streambyter both add a timestamp on receiving.

This implies when your keyboard sends delayed Note-Ons then this delay is recorded in Pianoteq.

You say the MIDI event delay depends on the cpu load of the Numa Compact 2??
That's very interesting. If it is true than other brands could be affected too. For example I am using my digital piano with switched-off internal speakers and just the MIDI-output connected with my PTQ-notebook.

This means the internal sound processing is always running in the background. Hmm, I will try to set it MIDI local-off now - maybe it will play "tighter" then!

Is it allowed to call it a "bug", when Numa's soundengine is always ON even with local-off??

Re: Mystery solved!

groovy wrote:

If I understand it correctly and try to put it in one sentence:
Your Numa Compact 2 sends MIDI Note-On events sometimes with an unexpected and audible high delay?

Yes, you are correct.


groovy wrote:

... normally there should be a minimum distance of 0.96 ms, but when your MIDI monitor just has a resolution of 1 ms then some events appear to be "at once" - nothing to worry about.

Ok, you might have a point here, but the fact is that the notes with the same time-stamp were always the notes that had a audible delay. How I checked:
- Play until I noticed a delay
- Immediately stop and keep my fingers on the keys so I could see which key were 
  pressed. If I had already played notes directly after the delay I checked what my
  previous chord was.
- Check the midi monitor and search for the notes I played that were delayed.
- Check the time-stamp ==> Always exactly the same.
- Start playing again and repeated this test for at least 4 times, same results every time. Two or more notes had the same time-stamp.


groovy wrote:

But why didn't I saw this with the Pianoteq midi viewer?

... accident, because not easy reproducible. Play long enough and you will catch those simultaneous events in PTQ's monitor.

Might be, but how long should I try? I've tried > six months now. Never saw anything strange in the midi event viewer.


groovy wrote:

... there is no "midi time-stamp" in realtime MIDI events sent from a MIDI keyboard. Your MIDI receiver/app adds a timestamp when a Note-On event is detected. Pianoteq's MIDI event monitor and Streambyter both add a timestamp on receiving.

This implies when your keyboard sends delayed Note-Ons then this delay is recorded in Pianoteq.

I'm having a hard time to agree on this one. Simply because the recordings are fine.
I played back the recorded midi data every time I heard a delay, but there was no delay on the recording. To me this implies there must be some sort of timing that is sent. If my recordings had the same delay as the live session I would fully agree with you.


groovy wrote:

You say the MIDI event delay depends on the cpu load of the Numa Compact 2??
That's very interesting. If it is true than other brands could be affected too. For example I am using my digital piano with switched-off internal speakers and just the MIDI-output connected with my PTQ-notebook.

Yes, I did not expected such an answer from Studiologic. I expected that, if I don't play any notes on a channel (or local control is off), the CPU does (almost) nothing.
So it was a bit confusing when I was told the sound engine is always active.
I do understand the decision to give the internal sound engine a higher scheduling priority than the midi engine, but to me the fact that they had to made this decision is proof of poor design.


groovy wrote:

This means the internal sound processing is always running in the background. Hmm, I will try to set it MIDI local-off now - maybe it will play "tighter" then!

This is certainly worth a try, but I hope your piano is designed way better than mine is


groovy wrote:

Is it allowed to call it a "bug", when Numa's soundengine is always ON even with local-off??

Yes, it's a bug for sure. I'm tempted to call it a bad design.

Last edited by MrRoland (05-11-2019 00:00)

Re: Mystery solved!

sjgcit wrote:

Nice bit of troubleshooting.  And very interesting info on how an apparently independent midi keyboard can throw things out of whack on a midi setup.  I'd never have expected that.  Hope this lets you get back to using Pianoteq (or whatever you choose) to make the sounds you want.

MrRoland wrote:

Studiologic support finally mentioned that the engine keeps running (and wasting cpu cycles) until you select a sampled instrument.

Ah yes, the joys of the "there's no need to shut down that process - it's doing no harm" logic I have heard so many times in my career.  First thing a programmer should be taught - release resources you are not using.

Thank you.
It was a mind boggling experience... I wasted a lot of time on this one,
but I also learned a lot.
As you mention, some engineers have strange habits. We, the customers, can complain, but it's hard to convince them... I hope Studiologic will try to improve their resource management, but I think they won't.

To me Pianoteq is the best software piano. The realism and flexibility is stunning.
But, I must admit, the Ravenscroft 275 on my Ipod is an excellent addition to my equipment. Not to replace Pianoteq, but to supplement it. I can make a setup on the Ipod and carry it in my pocket. As long there is a keyboard with usb I can plug it in and play. But for recordings and the ability to customize your sounds nothing compares to Pianoteq. So this means I will keep enjoying Pianoteq
(And I hope it will one day become available on Ios)

Last edited by MrRoland (04-11-2019 21:35)

Re: Mystery solved!

The only thing we know for sure in this thread is , that the MIDI protocol does not carry time-stamps. For example the code of a Note-On event consists of 3 bytes for channel, note and velocity (each byte framed in a start- and stop-bit so this event has 30 bit).

... here the dilemma starts: On which side of the cable is the origin of "the delay", sender or receiver?

A) Origin sender, then all delays are captured on the MIDI-recording of the receiver.

B) Origin receiver, then two hardwares (Ipod, Notebook), three OS (iOS, Linux, Windows) and two applications (Pianoteq, Ravenscroft) show the same delay symptoms.

Hypothesis A is tempting at first glance. But you can't hear the delays on the recording, only when playing realtime. - Just an idea: Is it possible that your delays are so small they can only be perceived as long as a reference is present (key touch)? but not without any reference when replaying the recording?

Or is your impression that the delays are so huge everybody could hear them on a recording?

Mystery evolves!

Re: Mystery solved!

Nice!

Re: Mystery solved!

Perhaps, veronica-brooke, another keyword here is Control.

MrRoland wrote:

Well, there's the catch...
Local control is off, but the engine is still running because midi receive will always be active.

I turned local control off, but nothing changed.

MrRoland, the engine runs possibly in anticipation it might receive MIDI messages (on sixteen {16} available channels) via the keyboard MIDI in.

Local control has no effect on incoming messages!  It happens on the connected device when connected by MIDI cable to your keyboard.

When you’ve a keyboard patch selected, one may assume it’s gotten priorities (from MIDI input circuits) to limited processor resources.

You likely free resources from possible bottle necks whenever you disengage Local control at any selected patch.  Thusly, delays are imperceptible or possibly avoided altogether.

Pianoteq 8 Studio Bundle, Pearl malletSTATION EM1, Roland (DRUM SOUND MODULE TD-30, HandSonic 10, AX-1), Akai EWI USB, Yamaha DIGITAL PIANO P-95, M-Audio STUDIOPHILE BX5, Focusrite Saffire PRO 24 DSP.

Re: Mystery solved!

Amen Ptah Ra wrote:
MrRoland wrote:

Well, there's the catch...
Local control is off, but the engine is still running because midi receive will always be active.

I turned local control off, but nothing changed.

MrRoland, the engine runs possibly in anticipation it might receive MIDI messages (on sixteen {16} available channels) via the keyboard MIDI in.

Local control has no effect on incoming messages!  It happens on the connected device when connected by MIDI cable to your keyboard.

When you’ve a keyboard patch selected, one may assume it’s gotten priorities (from MIDI input circuits) to limited processor resources.

You likely free resources from possible bottle necks whenever you disengage Local control at any selected patch.  Thusly, delays are imperceptible or possibly avoided altogether.


Let me explain how my Piano works:
- There are 4 zones which can be controlled independently:
  * 2 Midi zones (send only)
  * 2 audio (internal engine) zones (and midi receive)

- Every preset always contains these 4 zones, you cannot remove a zone.
- All zones can be layered / split / both.
- Every internal zone always has a patch selected.

The manual does not mention anything about local control (it is also not recognized as midi command).
I expected the on/of buttons for the four zones to be a real on/of like disable the whole channel (since they don't mention it as being for local control on/off). But it turned out to be for local control only.

In my preset, all zones were disabled (local control off) except 1 midi zone. But this didn't help.
If I had a option not to load any patch, I would have used it, but there is no such option.
There will always be two patches loaded, doesn't matter if you don't use the zone.
The only solution Studiologic support gave me is to select patches that are less resource hungry.
If there was an option to prevent the loading of a patch, they would have mentioned it I assume

To be sure I performed a factory reset on the piano so no "forbidden" patch combinations exist in the presets.
To bad all my user presets are gone, but at least everything runs stable now.

Last edited by MrRoland (06-11-2019 23:13)

Re: Mystery solved!

Thank you, MrRoland!

Pianoteq 8 Studio Bundle, Pearl malletSTATION EM1, Roland (DRUM SOUND MODULE TD-30, HandSonic 10, AX-1), Akai EWI USB, Yamaha DIGITAL PIANO P-95, M-Audio STUDIOPHILE BX5, Focusrite Saffire PRO 24 DSP.

Re: Mystery solved!

groovy wrote:

The only thing we know for sure in this thread is , that the MIDI protocol does not carry time-stamps. For example the code of a Note-On event consists of 3 bytes for channel, note and velocity (each byte framed in a start- and stop-bit so this event has 30 bit).

yes, my bad. I was digging into the midi protocol and found the section about the midi real-time controls like midi clock and midi time code. After your reply I discovered this only applies when using a sequencer or other device that sends the clock.

groovy wrote:

... here the dilemma starts: On which side of the cable is the origin of "the delay", sender or receiver?

A) Origin sender, then all delays are captured on the MIDI-recording of the receiver.

B) Origin receiver, then two hardwares (Ipod, Notebook), three OS (iOS, Linux, Windows) and two applications (Pianoteq, Ravenscroft) show the same delay symptoms.

Hypothesis A is tempting at first glance. But you can't hear the delays on the recording, only when playing realtime. - Just an idea: Is it possible that your delays are so small they can only be perceived as long as a reference is present (key touch)? but not without any reference when replaying the recording?

Or is your impression that the delays are so huge everybody could hear them on a recording?

Mystery evolves!

To me the delays sound like half a second or so, but I'm pretty biased because it has been bugging me for such a long time. So maybe the reality is that nobody else will hear it . I'm certain that it does feel / sound worse when I have the key touch as reference. I will test it next time when I'm playing with a friend of mine. I'll select a organ preset to force the delays to occur and see if he will notice it.

Last edited by MrRoland (07-11-2019 11:11)

Re: Mystery solved!

Thanks for your answers, MrRoland!

Interesting, that the Numa Compact 2 does not have the feature "local control off" (else Studiologic would have mentioned it in the MIDI implementation chart of this keyboard).

This MIDI feature decouples the internal keyboard from the internal sound engine, but leaves the MIDI output and input working.

As Amen Ptah Ra mentioned this feature would have been the simplest method to reduce cpu load caused by the internal sound engine. - Strange, that Studiologic doesn't support this long-time standard feature.

Btw I remember, that organ sounds can be quite demanding. When Organteq alpha came out, it overloaded my low-performance-netbook at that time, while normal Pianoteq worked. - Hmm, I should test Organteq alpha after this long time again ...

cheers

Re: Mystery solved!

groovy wrote:

Thanks for your answers, MrRoland!

Interesting, that the Numa Compact 2 does not have the feature "local control off" (else Studiologic would have mentioned it in the MIDI implementation chart of this keyboard).

This MIDI feature decouples the internal keyboard from the internal sound engine, but leaves the MIDI output and input working.

As Amen Ptah Ra mentioned this feature would have been the simplest method to reduce cpu load caused by the internal sound engine. - Strange, that Studiologic doesn't support this long-time standard feature.

Btw I remember, that organ sounds can be quite demanding. When Organteq alpha came out, it overloaded my low-performance-netbook at that time, while normal Pianoteq worked. - Hmm, I should test Organteq alpha after this long time again ...

cheers

Hi, I'm sorry if my post wasn't clear about that:
- The zone on/off buttons do function as local control on/of buttons, but turning local control off for all internal zones does not shutdown the
  engine and despite there are no notes (local or midi) sent to the engine, the CPU-load isn't reduced enough to get a stable midi stream.
  The only way to get midi to function properly is to change the preset to a less-demanding one....

Re: Mystery solved!

MrRoland wrote:

The zone on/off buttons do function as local control on/of buttons,

Are you sure? What, if they just mute the audio, but keep internal MIDI and audio processing coupled?

MrRoland wrote:

, but turning local control off for all internal zones does not shutdown the
  engine

... that's normal with "local control off" to keep the engine idle, because receiving MIDI input from external and synthesizing it to audio is wanted.

MrRoland wrote:

despite there are no notes (local or midi) sent to the engine, the CPU-load isn't reduced enough to get a stable midi stream.

When no notes are are sent to the engine, there should be no significant cpu-load from the engine. For example a webbrowser makes significant load only, when traffic is made (download sites, uploads etc.). Or compare Pianoteq's perf monitor --> cpu load just when playing notes.

MrRoland wrote:

, the CPU-load isn't reduced enough to get a stable midi stream.

Nobody has seen the CPU-load so far, neither the idle load nor the playing load

Last edited by groovy (07-11-2019 16:23)

Re: Mystery solved!

groovy wrote:
MrRoland wrote:

The zone on/off buttons do function as local control on/of buttons,

Are you sure? What, if they just mute the audio, but keep internal MIDI and audio processing coupled?

The quote below is a direct copy from a mail I received from Studiologic:

the UPPER and LOWER buttons are like a LOCAL OFF control of the related internal parts.

The ZONE A -B buttons (same 2 buttons below the display) are controlling the MIDI ZONES separately, allowing to send or not notes and controls on the programmed MIDI Channels from MIDI Outs phisical or over USB

The internal UPPER and LOWER parts could be played via MIDI on the related CH1 and CH2 even if the LOWER and UPPER buttons are off, meaning that the local control is disabled and not the Sound module via MIDI.

groovy wrote:
MrRoland wrote:

despite there are no notes (local or midi) sent to the engine, the CPU-load isn't reduced enough to get a stable midi stream.

When no notes are are sent to the engine, there should be no significant cpu-load from the engine. For example a webbrowser makes significant load only, when traffic is made (download sites, uploads etc.). Or compare Pianoteq's perf monitor --> cpu load just when playing notes.

You're right about one thing: there should be no significant cpu-load from the engine. The reality seems to point in an other direction.
I'm still in contact with Studiologic and asked for more info about the cpu/dsp load when (almost) idle. Hopefully they are willing to share some more details.

Re: Mystery solved!

Yes, after that e-mail it seems to be local control off what they implemented undocumented.

Because there are two buttons two MIDI channels have to be set local off to disconnect the keyboard completely from the internal sound modules.