Topic: Feature request Chart for Archived MIDI - see how much you've played

As title. Would be great if I could see with one look how much hours / notes I play daily on each individual day.
Maybe as a graph? Would be great to also see it at a glance. Year for year, month for month.

Re: Feature request Chart for Archived MIDI - see how much you've played

interesting idea.  on linux it wouldn't be so bad to hack something together out of python/perl and gnuplot to do this, but maybe that's not the answer you're after ...

i might try it at some point to see what happens.

Re: Feature request Chart for Archived MIDI - see how much you've played

i managed to do it.  not sure how illuminating it is but it was fun to do.  i don't see any way to upload it to the forum though ...

Re: Feature request Chart for Archived MIDI - see how much you've played

budo wrote:

i managed to do it.  not sure how illuminating it is but it was fun to do.  i don't see any way to upload it to the forum though ...

Could you showcase it? I'm on MacOS here
But if enough people like it, it might just be done by Modarrt for the next minor update.

Re: Feature request Chart for Archived MIDI - see how much you've played

Defenz0r wrote:
budo wrote:

i managed to do it.  not sure how illuminating it is but it was fun to do.  i don't see any way to upload it to the forum though ...

Could you showcase it? I'm on MacOS here
But if enough people like it, it might just be done by Modarrt for the next minor update.

Push

Re: Feature request Chart for Archived MIDI - see how much you've played

Fun idea, but i'm not entirely sure there would be a huge fan base for it in this community - and therefor it might not end up very high on the Modartt priority to-do list...

budo wrote:

not sure how illuminating it is

That would be my worry too - to me (personally) it would be akin to tracking everything trackable on your road bike for a day or two (just because it's possible), publish it proudly to all friends on facebook and then loose interest for lack of further purpose (other then fun).

In the mean time: have you seen that if you instruct PT to record everything you play, you'll end up with a list of midi files the names of which contain the date and day, the number of notes played, and the time it took you to play them. You could collect that data in a spread sheet and calculate all sorts of things - a little more effort and typing, but the same result. The function is a little hidden:

File >  Recently played on the keyboard > MIDI archive settings
(or click the demo name in the section of the User Interface, and choose the bottom option)

Last edited by thiesdewaard (26-01-2021 15:43)

Re: Feature request Chart for Archived MIDI - see how much you've played

I have a quick and dirty solution in my raspi4 to know how much and how fast I play every day:

Just paste this code (works for pianoteq on linux)

find $HOME/.local/share/Modartt/Pianoteq/Archive -name "*.mid" | awk '{notes[$1] += $4; seconds[$1] += $6} END{for (i in notes) print i, notes[i]" notes" , seconds[i]/60" minutes", notes[i]/seconds[i]" nps"}' | awk -F"/" '{print $11}' | sort

The output it's like this

2021-01-05 3669 notes 15.9333 minutes 3.83787 nps
2021-01-06 6537 notes 26.1167 minutes 4.17167 nps
2021-01-07 26628 notes 127.633 minutes 3.47715 nps
2021-01-08 5937 notes 29.35 minutes 3.37138 nps
2021-01-09 32459 notes 148.867 minutes 3.63401 nps
2021-01-10 35140 notes 143.667 minutes 4.07657 nps
2021-01-11 17872 notes 71.1833 minutes 4.1845 nps
2021-01-12 24780 notes 92.6333 minutes 4.45844 nps
2021-01-13 26301 notes 109.683 minutes 3.99651 nps
2021-01-14 4796 notes 21.45 minutes 3.7265 nps
2021-01-15 19746 notes 75.6 minutes 4.35317 nps
2021-01-16 38128 notes 131 minutes 4.85089 nps
2021-01-17 7857 notes 32.75 minutes 3.99847 nps
2021-01-18 28448 notes 97.5 minutes 4.86291 nps
2021-01-19 29820 notes 108.717 minutes 4.57152 nps
2021-01-20 20318 notes 75.75 minutes 4.47041 nps
2021-01-21 26757 notes 153.7 minutes 2.90143 nps
2021-01-22 12677 notes 62.5 minutes 3.38053 nps
2021-01-23 18173 notes 85.5667 minutes 3.53974 nps
2021-01-24 25681 notes 164.333 minutes 2.60456 nps
2021-01-25 21339 notes 142.683 minutes 2.49258 nps
2021-01-26 11624 notes 62.1333 minutes 3.11803 nps



Some day I sould do something more refined in python .....

Re: Feature request Chart for Archived MIDI - see how much you've played

Push

i'd be happy to share the image, i just don't see how to do it in the forum ...

maybe someone could explain how to upload an image?

Re: Feature request Chart for Archived MIDI - see how much you've played

dcardenosa wrote:

I have a quick and dirty solution in my raspi4 to know how much and how fast I play every day:

Just paste this code (works for pianoteq on linux)

find $HOME/.local/share/Modartt/Pianoteq/Archive -name "*.mid" | awk '{notes[$1] += $4; seconds[$1] += $6} END{for (i in notes) print i, notes[i]" notes" , seconds[i]/60" minutes", notes[i]/seconds[i]" nps"}' | awk -F"/" '{print $11}' | sort

Some day I sould do something more refined in python .....


How todo that on macOS? I've only made it for a month each (2021-1) with CyberChef and LibreOffice.
I've used RegEx

You may use: BBCode Images Smilies -> I used BBCode for images here.

https://i.imgur.com/UbjDKNf.png

Last edited by Defenz0r (27-01-2021 14:47)

Re: Feature request Chart for Archived MIDI - see how much you've played

thanks, i see how to do it now (what i was missing was that the image had to be hosted somewhere online ... there doesn't seem to be a way to upload local files).

this is what i got from analyzing 2020 (and adding up the total notes and time played per day)


https://thumbs2.imgbox.com/ed/b4/1R5hd7mB_t.png

Re: Feature request Chart for Archived MIDI - see how much you've played

budo wrote:

thanks, i see how to do it now (what i was missing was that the image had to be hosted somewhere online ... there doesn't seem to be a way to upload local files).

this is what i got from analyzing 2020 (and adding up the total notes and time played per day)


https://thumbs2.imgbox.com/ed/b4/1R5hd7mB_t.png

Can you share the script? I'm wanting to work on it and put it on my server.

Cause I might be too long busy in doing it myself, I might pay someone for quick & dirty work. Want to get it done so it works with https://www.chartjs.org/

Thinking more of a month/day basis. Like a calender. But that is to be decided yet.

Last edited by Defenz0r (27-01-2021 17:36)

Re: Feature request Chart for Archived MIDI - see how much you've played

thiesdewaard wrote:

Fun idea, but i'm not entirely sure there would be a huge fan base for it in this community - and therefor it might not end up very high on the Modartt priority to-do list...

budo wrote:

not sure how illuminating it is

That would be my worry too - to me (personally) it would be akin to tracking everything trackable on your road bike for a day or two (just because it's possible), publish it proudly to all friends on facebook and then loose interest for lack of further purpose (other then fun).

Information regarding exercise efforts can be extremely important for building and maintaining a habit.  The website Strava.com has built a valuable business off of this, giving runners and cyclists a place to upload data (collected on gps watches or cycling computers) regarding every run or ride they do.  Having this record is really helpful for all sorts of reasons: there's no uncertainty regarding what you've done in the past, you can motivate yourself ("I'm going to try to do more than last week), there's an incentive to keep streaks going ("I've run every day for the last two weeks and I want to keep that streak going"), and more.  Way more.

Yes, something like Strava makes it public and has a social media aspect.  But that's not necessary at all, and many people use it despite not liking the social aspects Strava.  (There are of course ways to do this entirely in private, using other apps, but many people find that Strava's public nature helps them keep getting out there for regular running and cycling; not a bad thing in my opinion.)

Anyway, I've tried several times to clock and record all my piano practice time.   I'd like to know it.  I think knowing that would be helpful.  But every time I've tried I've failed.  Never was able to develop the habit of logging all my time.

I'd never thought about it before, but that archived group of midi files in Pianoteq is actually a good source of info to create a record that gives a pretty good indication of how much you've practiced.  It doesn't exactly indicate the time spent sitting at a piano keyboard.  But it would not be far off for me. And it's quite easy to process that list of files in a Python script (or some other scripting language) to convert the list of filenames into a useful form, both textual and graphic.  Whether it's in Python or some other language, there are ways to make it easy to execute that script on a Mac, Windows, or Linux PC.

So I'll look forward to checking out any script that's made public for this.  Or I'll end up writing my own and offering it up here.

Re: Feature request Chart for Archived MIDI - see how much you've played

> So I'll look forward to checking out any script that's made public for this.  Or I'll end up writing my own and offering it up here

Thanks! Might be possible I will distribute one, too.
But if you try, may ...

Make sure it will be possible to run on a Server, like Ubuntu or so. Because else, it might be harder to maintain.
Running it locally could be easier, but I'm uncertain if thats as good.
Having stuff displayed in the browser is just great...

My idea is to simply sync with MEGA, fetch the data regularly with MEGAcmd on my Ubuntu Server, and having the graphs / charts refresh accordingly.

What might be inspiring are those graphs https://cultris.net/
But the solution should not load so long (laughs)

Last edited by Defenz0r (27-01-2021 18:51)

Re: Feature request Chart for Archived MIDI - see how much you've played

here's the script i used (perl).  it was quick and dirty, so please no code bashing   (well, if you have some advice that's great.) i'm on a linux box.  there are two perl modules i used for convenience, Time::Piece and File::Finder.

my midi files are sitting under this in directories ./2021, ./2020, etc., with subdirectories for each month, just as PTQ stores them in the midi archive.  in particular the months are 0-padded 2-digit strings.

the script writes to stdout and produces three columns.  the first is day-of-year, the second the notes played on that day, and the third is the time in seconds on that day.  i loaded the output into gnuplot to produce the graph.

#!/usr/bin/perl

use Time::Piece;
use File::Finder;
use autodie;
use v5.10;

die "usage: $0 year\n" if @ARGV < 1;

my ($year) = @ARGV;

my @midifiles = grep {/\.mid/} File::Finder->in("./$year/");

my %playingdata = ();
for my $f (@midifiles) {
  my $date = "";
  my $notes = "";
  my $seconds = "";
  if ($f =~ m|\./$year/\d\d/([\d-]+) \d+ [()\w]+ (\d+) notes, (\d+) seconds|) {
    ($date, $notes, $seconds) = ($1, $2, $3);
  }
  my $dayofyear = Time::Piece->strptime($date, "%Y-%m-%d")->yday();
  $playingdata{$dayofyear}{"notes"} += $notes;
  $playingdata{$dayofyear}{"seconds"} += $seconds;
}

for (sort {$a <=> $b} keys %playingdata) {
  say "$_ $playingdata{$_}{'notes'} $playingdata{$_}{'seconds'}";
}

Re: Feature request Chart for Archived MIDI - see how much you've played

>here's the script i used (perl).  it was quick and dirty, so please no code bashing   (well, if you have some advice that's great.) i'm on a linux box.  there are two perl >modules i used for convenience, Time::Piece and File::Finder.

I mean, that's all good, but could you parse some graphs through the Commandline/Shell, using this as template? Perl or Python might be a good basis, cause you can run it anywhere, I guess.
https://jsfiddle.net/Lso6nmu4/

You could basically put this as HTML code, and at the point you want to use values, add them. The CSS is completely optional.
So you are generating a HTML file, that you can then Host on Port 80.

What do you think?
I mean, it could run on a webserver this way.

Last edited by Defenz0r (28-01-2021 02:04)

Re: Feature request Chart for Archived MIDI - see how much you've played

Defenz0r wrote:

Make sure it will be possible to run on a Server, like Ubuntu or so. Because else, it might be harder to maintain.
Running it locally could be easier, but I'm uncertain if thats as good.
Having stuff displayed in the browser is just great...

My idea is to simply sync with MEGA, fetch the data regularly with MEGAcmd on my Ubuntu Server, and having the graphs / charts refresh accordingly.

I don't know what MEGA is.  But the problem with having something on a webserver to do this is that accessing a server from your webbrowser won't be able to access the local filesystem and get the names of the all the files, which are what contain the data needed for the analysis.  Users could have some local app that processes the filenames and gives a lump of text to upload to a server, but it's barely any harder to build a local app that does all the analysis without having a server involved.  So I think the server might actually end up complicating things for users, not simplifying.

Re: Feature request Chart for Archived MIDI - see how much you've played

Defenz0r wrote:

You could basically put this as HTML code, and at the point you want to use values, add them. The CSS is completely optional.
So you are generating a HTML file, that you can then Host on Port 80.

What do you think?
I mean, it could run on a webserver this way.

Looks good to me.  If not running on a local server can this actually run in the browser?  Does sandbox include ability to access the local filesystem like that?  I didn't think it did,   (but I obviously know little about web development.)

Re: Feature request Chart for Archived MIDI - see how much you've played

You could basically put this as HTML code, and at the point you want to use values, add them. The CSS is completely optional.
So you are generating a HTML file, that you can then Host on Port 80.

I'm going to try today again.

You can access the local files. You are just going to overwrite the html file lets say each hour. The files either are already on the server because pianoteq runs on it or because they are synced from client to server regularly.

The sync is actually the easiest.

You would just copy the html code and append your data. Top to bottom. Or bottom to top.
Http-server from npm is good suited for that task of hosting either locally or on the web. On the web you would simply add nginx for https

**Edit** Working on it, think I can do it this weekend or so.

Last edited by Defenz0r (28-01-2021 12:06)

Re: Feature request Chart for Archived MIDI - see how much you've played

Have got something to work, with maximum effort. It auto adapts, to the existing data. So if you have recordings for more months, it will display that, as well.
Takes a path of your choice, gives you back for instance

["2020-12", "2021-1"]
[20.081944444444446, 19.486666666666668]
[241454, 212237]

The fetcher is written in Rust, will upload it later. I may give it another name, too.
The HTML is an adapted HighChart one.

For people who want it on the same machine, you can just pass the stdout like so: ./matcher /root/MIDI/ >/root/www/stat/out.txt

I let it run all 10 minutes, like so

1. apt-get install cron
2. crontab -e
3. edit with nano
4. 10 * * * * /root/tool/./matcher /root/MIDI/ >/root/www/stat/out.txt
(first argument is the input folder, then basically stdout to your folder where your HTML is in)

To note is that, I do not yet know what I should do with the monthly played notes value, to me, it is pretty useless right now.

The the HTML file I have in the directory, where out.txt is, loads the Highcharts graph, from the .txt, thats within the same folder, upon entering.

Played a bit around, and this is refreshing all 10 minutes the JSON for input ->
https://stat.def0.de/highcharts-bar-w-sub.html
https://stat.def0.de/highcharts-line.html
https://stat.def0.de/highcharts-column.html

Once I have everything at hand, I may should put something on github or so, with screenshots and all.

Last edited by Defenz0r (29-01-2021 19:40)

Re: Feature request Chart for Archived MIDI - see how much you've played

I wonder if all this syncing and server stuff is necessary, or making things easier.

E.g., it's pretty easy to create a file containing a list with all the midi playing data:

To do this on a Windows PC, in a terminal window, just enter the command:

>  dir c:\users\[your user name]\AppData\Roaming\Modartt\Pianoteq\Archive\*.mid /s > mylist.txt

This will create a file with a list of all the midi filenames (i.e., all the data), and it will have the name 'mylist.txt'.

That file can then be processed to create a graph, either by uploading to a server app, or processing with some fairly simple app on your pc. 

The 'dir' command itself can be put into a simple batch file so a user can run it simply by clicking an icon on their desktop.

Last edited by hesitz (29-01-2021 20:28)

Re: Feature request Chart for Archived MIDI - see how much you've played

-> That file can then be processed to create a graph, either by uploading to a server app, or processing with some fairly simple app on your pc.

Sure, but needs much postprocessing then. Not goint todo that for each day / month. Programmatically, you have the flexibility.

I don't want to do that regularly. It just works. But I will keep polishing it a bit beforehand. You do not need a Server, but it's recommended. Even a local one does.
I see the progress as soon as I want. Planning to set daily goals to meet.

@dcardenosa, I used parts of your command, to reform a valid one for the needed format.

https://github.com/zDEFz/PTQ-Stat/blob/main/README.md

https://stat.def0.de/
^ working sample.

Notes played seem pretty useless to me. Should work out the readme a bit. But it's pretty straight forward.

With that, and the code above, you can basically make it yourself.

Last edited by Defenz0r (30-01-2021 21:39)

Re: Feature request Chart for Archived MIDI - see how much you've played

Actively working on it.

I just made a update in the repo https://github.com/zDEFz/PTQ-Stat

To note is, that this is my first draft, it works properly for me since days, but I still make some changes.
As said, you can see how it looks here https://stat.def0.de/

There is a lot to rework in the codebase, it's not yet clean and a lot of debug features are enabled.
It's hacked together.