budo wrote:

respectfully i must disagree.  or at least i have to say, these arguments are not convincing to me.

And I, also respectfully, invite you to read the technical paper by the authors of the method called Onsets and Frames, and you will see why I raised my doubts that I maintain. The deconvolution techniques that are at the core of the method, are not capable of making that distinction, in other than in notes in clear sonic separation of other notes. And even then I believe the separation could only be made by internal comparison, ie by comparing the sonic signature of the same note played in several different moments.

It is much more possible (although it would amount to the complexity of the best AI programming) to calculate pedaling through the estimation of finger position and movements.

no problem, we all have respect here .  i have read the paper.  i'm not advocating doing what they did, or tweaking it, but rather something much more vague: developing a new magenta model.  i have no idea if it's possible, and i'm not an expert, but i also am not convinced that progress couldn't be made.  if one had told me 10 years ago that onsets and frames would exist, i would have been very suprised (actually i would have been just as surprised at the existence of Pianoteq, but i guess that's another topic).  any model would be great, whether based on estimation of finger position or whatever else one thinks is the right way forward.  i was just drawn more to a purely sonic model as a first approximation.  but i probably will never take it on anyway ... too many other incomplete projects.

respectfully i must disagree.  or at least i have to say, these arguments are not convincing to me.

And I, also respectfully, invite you to read the technical paper by the authors of the method called Onsets and Frames, and you will see why I raised my doubts that I maintain. The deconvolution techniques that are at the core of the method, are not capable of making that distinction, in other than in notes in clear sonic separation of other notes. And even then I believe the separation could only be made by internal comparison, ie by comparing the sonic signature of the same note played in several different moments.

It is much more possible (although it would amount to the complexity of the best AI programming) to calculate pedaling through the estimation of finger position and movements.

I very much doubt that a model for automatic pedal detection can ever be created.

I totally agree with you...

respectfully i must disagree.  or at least i have to say, these arguments are not convincing to me.

the point is the way these models work.  they are doing absolutely nothing like the kind of quantitative reasoning a human might do to evaluate whether the pedal has been depressed.  all they do is try to design an algorithm by analyzing lots and lots of data that has already been marked as "pedal depressed" or "pedal not depressed".  the software makes a guess, scores itself, and then improves its guess.  the mathematics behind the process is very interesting and can't really be gone through here, but the point is there is a well-defined way for the software to modify its algorithm based on its performance.  after many many iterations of this, assuming the model is designed well, it will perform very well on the training set, and also on new input it's never seen before.  how it's "thinking" is completely different from how a human actually thinks.

already a lot of the sonic ambiguity mentioned is present in solo piano and is handled very well by onsets and frames.  for instance, how can we distinguish the fundamental of a note and some higher overtone?  the tone of a piano note is very different when it's played loudly or softly.  how do we distinguish that?  we don't ... we just train the model on data and let it sort it out internally.  (to be fair, one weakness of the current model is detecting velocity.  the midi velocity values are reasonable but not close to a human performance, imo.  still it's doing a remarkable job.)

of course the only way to demonstrate success is succeed.  i'd like to try at some point, but i don't know when i'll have time.  maybe someone else will try

teacue wrote:

@ YvesTh
Very interesting points, thank you for this.
A not easy task and probably even more difficult for a non-pianist.

@ budo
A new magenta model would be indeed welcome.

I very much doubt that a model for automatic pedal detection can ever be created.
From a sonic perspective, it is impossible to differentiate if a note is sustained by the key or by the pedal (yes, the resonances are different if other notes are played at the same time, but I bet my money that the machine learning model focuses only on fundamental frequencies or otherwise the analysis would be incredibly complex). Moreover, you have the sustain pedal that lifts all dampers, and the sostenuto pedal that prevents lifted dampers from dropping. So, when a note is held, it is impossible to know which of the three methods is being used:  key, sustain pedal, sostenuto pedal.

The only way is applying a "feasibility" test: each finger is doing what? Does this requires more than 10 fingers?

I totally agree with you...

@ YvesTh
Very interesting points, thank you for this.
A not easy task and probably even more difficult for a non-pianist.

@ budo
A new magenta model would be indeed welcome.

I very much doubt that a model for automatic pedal detection can ever be created.
From a sonic perspective, it is impossible to differentiate if a note is sustained by the key or by the pedal (yes, the resonances are different if other notes are played at the same time, but I bet my money that the machine learning model focuses only on fundamental frequencies or otherwise the analysis would be incredibly complex). Moreover, you have the sustain pedal that lifts all dampers, and the sostenuto pedal that prevents lifted dampers from dropping. So, when a note is held, it is impossible to know which of the three methods is being used:  key, sustain pedal, sostenuto pedal.

The only way is applying a "feasibility" test: each finger is doing what? Does this requires more than 10 fingers?

With pedal sustain addition at the begining and various corrections :

https://forum.modartt.com/uploads.php?f...idi%29.mid
With velocity curve :
Global Velocity = [0, 26, 46, 72, 102; 0, 23, 44, 79, 127]
Duke Ellington was playing on a Steinway for this recording.
Pictures on this link :
https://ellington.se/2021/02/25/ellingt...elas-1966/

You both did not answer my question about dynamic.
Any idea about this?

You are right, on my test the dynamic needs to be reworked. I think that the choice of the pianoteq instrument used is very important, to that it is necessary to rework the velocity curve for the reading of the midi file and even sometimes to modify individually the velocity of some notes.

@ budo
A new magenta model would be indeed welcome.

You both did not answer my question about dynamic.
Any idea about this?

As a non-pianist I would like to learn in this context how to restore the sustain pedal.
Has someone some tips?

@YvesTh gave a great answer with everything i wanted to say about this.  i just wanted to add that a longer-term way to solve the problem would be to develop a new magenta model that does this.  it would take as input the same data as onsets and frames (wav file) and produce a guess about application of the sustain pedal.  it could be trained on lots of audio snippets, the same as onsets and frames.  it should actually be easier to do this than what onsets and frames already does, because it's training for a binary state (pedal is on vs pedal is off).  of course it could be made more complicated because there are lots of coloristic effects one can do with the pedal that go beyond on/off, but at least this would be a start.

As a non-pianist I would like to learn in this context how to restore the sustain pedal.
Has someone some tips?

If the notes held are unplayable by the pianist (finger spread) it means that the sustain pedal has been used. Otherwise you have to listen carefully to the original file to see if you can hear the particular resonance of the piano with the pedal depressed. The problem is that pianists often use the pedal on short sequences of notes that are very difficult to determine, and the time and speed of releasing the pedal is just as important as the time and speed of depressing it. I have on the "Duke" file a lot of trouble to determine everywhere these parameters....
For classical music you can read the score indications but for a jazz improvisation you have only your ears...

Translated with www.DeepL.com/Translator (free version)

As a non-pianist I would like to learn in this context how to restore the sustain pedal.
Has someone some tips?

As one can clearly hear in YvesTh transcription much of the dynamic get lost.
Beside of editing each note manually what could be done to get the whole dynamic range?
I tried editing the velocity curve but found out that it is not easy to achieve good results.

Original file : Duke Ellington solo piano at Goutelas in 1966

https://forum.modartt.com/uploads.php?f...ait%29.mp3

Pianoteq File "Steinway B" with midi file from magenta ( with some corrections).

https://forum.modartt.com/uploads.php?f...ess%29.mp3

I will try to make many other necessary corrections.

Main problems on the midi file.
Many excess notes had to be removed.
Low notes coming from Duke's grunts, high notes coming from very strong harmonics, high notes coming from clapping, the software not being able to detect the sustain pedal, the notes are extended individually and therefore we don't have the sympathetic resonance of the other strings with pianoteq. The ideal would be to restore the pedal effects in the midi file (I added some quickly). Some missing notes too. It's will be a big work to do but very interesting.
The original file is about 2 x 10 minutes long.