Offline version of the Magenta Audio to Midi transcription tool

Thank you for this very good idea.
I tested it on an extract of a rare concert of Duke Ellington alone at the piano ( Duke at Goutelas, vinyl 1966).  I did a test with a melodyne demo and the result is impressive on some passages.  It is necessary however to rework the midi file a lot, first of all the software does not find the use of the sustain pedal which it is thus necessary to add, Duke use it a lot. Some of Duke's growls or noises are considered as notes and some harmonics too. And sometimes some notes are considered as harmonics and are deleted by the software. But Melodyne allows you to recover these notes. I did another test with magenta but I could only do it on a very short extract and I will try again. I'd love having the Duke playing for me on my Pianoteq but it's I think hours and hours of work... 

Translated with www.DeepL.com/Translator (free version)

Here is an example of an audio to midi transcription of the Magenta tool.
https://forum.modartt.com/uploads.php?f...anoteq.mp3

I choosed a Piano Roll Recording of the "Pavane pour une infante défunte" played by Ravel himself:
https://www.youtube.com/watch?v=7ASYm3K_PwM

It may not be an ideal example as it is a transcription of a transcription, but well, I love this interpretation.

I did not change anything to the Midi data of the transcription because I am not a pianist and also it should be an example of what this tool is doing on its own.
The Pianoteq preset is HB Steinway D Classical with "Long Plate" as reverb.

I am just a humble, low skilled piano player, doing it only for my own enjoyment - but the even older technique, hand played piano rolls, often by the composers themselves, are kind of a treasure. Even if they lack dynamics "per key" or have no dynamics at all.

i'm glad onsets and frames is getting more love ... they did a great job with it and their work really is a significant achievement. 

i myself installed it (on a linux system) using conda:

https://docs.conda.io/en/latest/

it should also work on Mac/PC.  there are other models in the magenta project that are also very fun to play with.  one, for instance, tries to write a four-voice chorale given a short melody.  another model attempts to generate improvisations based on sample music. 

regarding some of the questions asked above, the software was developed to produce midi of solo piano recordings.  there's a lot that goes into it, but the main point (in this kind of machine learning) is that the computer develops its own algorithm to process the wav files and produce midi by training itself on a large dataset.  this dataset has lots of snippets of pairs (audio, corresponding midi).  the model runs repeatedly on the dataset and teaches itself by assigning a score to its most recent attempt and modifying the algorithm to try to do better.  it needs many generations of this before it can reliably handle audio data it hasn't seen before.  this means it can't handle other kinds of music processing problems (like separating instruments, or analyzing music other than solo piano). 

on the other hand, just because it was designed to run on solo piano, it doesn't mean one can't try other things to see what happens   i did this a while ago; it's a lot of fun to feed it music made from different instruments to see what happens.  here are three examples:

https://hearthis.at/budosaurus/5thwav/

https://hearthis.at/budosaurus/01-black-dogwav/

https://hearthis.at/budosaurus/impwav/

Indeed

Great examples

that was a very impressive installation description from teacue  

regarding the length of the wav file, on my machine the tool can handle fairly long performances but certainly not arbitrarily long.  if the program requests too much memory from the system as it analyzes the file, it kills itself.  i've tried to figure out a workaround but no luck so far.  20-40 min performances seem ok, but beyond that it dies.

Thanks, budo,  Clever demonstrations.

,,,and one of the only times that I've heard Led Zeppelin mentioned on the PIanoteq forum (b.t.w., have you guys heard the new Alison Kraus and Robert Plant album, Raise the Roof?

you can just put your Duke Ellington video on youtube and then put the full url into the text of the post.  the forum software seems to be clever enough to handle everything.

Thanks, Yves.  I have gone from 'wowed by the technology' to now having glimpsed behind the curtain and seen all the work that must go on behind the scenes.  Clearly, it's not 'turn-key' technology the way that shooting a digital photo of an old print or slide is, where auto-exposure makes that pretty easy now-a-days.  I am sure that an 'auto-musicality' function can't be too far behind!  ;-)

If the notes held are unplayable by the pianist (finger spread) it means that the sustain pedal has been used. Otherwise you have to listen carefully to the original file to see if you can hear the particular resonance of the piano with the pedal depressed. The problem is that pianists often use the pedal on short sequences of notes that are very difficult to determine, and the time and speed of releasing the pedal is just as important as the time and speed of depressing it. I have on the "Duke" file a lot of trouble to determine everywhere these parameters....
For classical music you can read the score indications but for a jazz improvisation you have only your ears...

Translated with www.DeepL.com/Translator (free version)

@ YvesTh
Very interesting points, thank you for this.
A not easy task and probably even more difficult for a non-pianist.

@ budo
A new magenta model would be indeed welcome.

You both did not answer my question about dynamic.
Any idea about this?

Midi file extract :

With pedal sustain addition at the begining and various corrections :

https://forum.modartt.com/uploads.php?f...idi%29.mid
With velocity curve :
Global Velocity = [0, 26, 46, 72, 102; 0, 23, 44, 79, 127]
Duke Ellington was playing on a Steinway for this recording.
Pictures on this link :
https://ellington.se/2021/02/25/ellingt...elas-1966/

I very much doubt that a model for automatic pedal detection can ever be created.
From a sonic perspective, it is impossible to differentiate if a note is sustained by the key or by the pedal (yes, the resonances are different if other notes are played at the same time, but I bet my money that the machine learning model focuses only on fundamental frequencies or otherwise the analysis would be incredibly complex). Moreover, you have the sustain pedal that lifts all dampers, and the sostenuto pedal that prevents lifted dampers from dropping. So, when a note is held, it is impossible to know which of the three methods is being used:  key, sustain pedal, sostenuto pedal.

The only way is applying a "feasibility" test: each finger is doing what? Does this requires more than 10 fingers?

respectfully i must disagree.  or at least i have to say, these arguments are not convincing to me.

the point is the way these models work.  they are doing absolutely nothing like the kind of quantitative reasoning a human might do to evaluate whether the pedal has been depressed.  all they do is try to design an algorithm by analyzing lots and lots of data that has already been marked as "pedal depressed" or "pedal not depressed".  the software makes a guess, scores itself, and then improves its guess.  the mathematics behind the process is very interesting and can't really be gone through here, but the point is there is a well-defined way for the software to modify its algorithm based on its performance.  after many many iterations of this, assuming the model is designed well, it will perform very well on the training set, and also on new input it's never seen before.  how it's "thinking" is completely different from how a human actually thinks.

already a lot of the sonic ambiguity mentioned is present in solo piano and is handled very well by onsets and frames.  for instance, how can we distinguish the fundamental of a note and some higher overtone?  the tone of a piano note is very different when it's played loudly or softly.  how do we distinguish that?  we don't ... we just train the model on data and let it sort it out internally.  (to be fair, one weakness of the current model is detecting velocity.  the midi velocity values are reasonable but not close to a human performance, imo.  still it's doing a remarkable job.)

of course the only way to demonstrate success is succeed.  i'd like to try at some point, but i don't know when i'll have time.  maybe someone else will try

And I, also respectfully, invite you to read the technical paper by the authors of the method called Onsets and Frames, and you will see why I raised my doubts that I maintain. The deconvolution techniques that are at the core of the method, are not capable of making that distinction, in other than in notes in clear sonic separation of other notes. And even then I believe the separation could only be made by internal comparison, ie by comparing the sonic signature of the same note played in several different moments.

It is much more possible (although it would amount to the complexity of the best AI programming) to calculate pedaling through the estimation of finger position and movements.