Can the latency be too short?

This is an interesting topic and I did a similar "discovery" as yours a while ago. I work as a piano teacher and this spring I was teaching online because of covid-19. I've always gone for lowest latency possible but this situation forced me to increase latency due to performance issues with my computer when streaming the sound through google meet and use webcams and other necessary programs at the same time. Surprisingly I found out that a latency around 5ms felt more "right" when playing with headphones and give me a more acoustic feel to the playing experience. The slightly higher latency also makes the keyboard feel a little heavier, especially when it comes to the percieved dynamic weight, which is something I've been wanting with my Yamaha N1. So I guess we are at a point where latency is no longer an issue.

Old topic, but some new observations that possibly fit here:

I'm testing another DP at the moment. The Korg D1 via 5-pin MIDI-out and a Roland UM-One Mk2 USB-interface to my lightweight Linux-laptop. The D1 with the Roland interface seems to have a shorter overall latency than my former setup. At the same time it has a louder mechanical key noise than I'm used to. This has two consequences in my opinion. The pianosound "feels" a tad too early and the attack of Pianoteq is unpleasently covered / overlapped by the acoustic key noise. I could face it with sheer volume boost like a real piano does, but I tried another strategy ...

Increasing the latency of Pianoteq.

Normally I used that laptop with 64 samples Buffersize at 44100 Hz. Given that Pianoteq 9 has latency of 3 times the buffer size [1] this means a latency of:

64 / 44100Hz * 3 = 4.4 ms

Now I have chosen 128 samples Buffersize at 48000 Hz. This means a slighly higher latency:

128 / 48000Hz * 3 = 8.0 ms

In other words the latency is 3.6 ms higher now.

It is not too much to be unacceptable in my opinion, but distant enough to better separate the mechanical key noise from Pianoteq's attack sound.

[1] Pianoteq 9 README_LINUX.txt
"* with Jack or Alsa, use buffer sizes that are multiple of 64, as Pianoteq prefers those sizes. With the Alsa driver, the latency is 3 times the buffer size (a buffer of 64 samples at 44100Hz gives a latency of 4.3 ms)."

It is a fascinating topic .  Now there is a difference between an acoustic grand piano and a digital piano :
in a digital piano the overall latency is essentially independent of note- on velocity . On the other end,  in an acoustic grand, academic research , the most known being Anders Askenfelt and A. G. Jansson documents , shows that delta time between key bottoming out and sound being heard  depends on dynamics,  due to speed of the hammer travelling on its own after escapement point;

at ppp , the hammer travels slowly and the sound arrives after the finger reaches the bottom of the key ; at fff its the opposite , the hammer travels very fast and sound is heard  before key touches the keyed bottom.   Depending on each grand piano , there is a specific point ( between mf and f ) where both sounds coincide.

In digital pianos, latency with internal sound is very low. Kawai has introduced a  specific parameter in virtual technical  called 'hammer delay' to specially delay sound for low velocities .  Given the fix nature of latency in digital keyboards and in absence of such parameter usable for external libraries such as pianoteq , the ideal would be to set up the latency low enough so that at fff , the sound is heard before key bottoms out  and to include a script in a DAW or VST host software that artificially adds delay for low velocities . 
One other variable to consider  is that  some DP makes more noise at key release that key bottoming out.  Anyway  just food for thoughts...

You are correct . This is the correct normal behaviour . I got mixed up in my reply .but the variable delta is what matters here and that’s what is difficult to reproduce with dp

That's indeed one of the limitations of many DP's. A few of them with very low friction are capable of  reaching the bottom sensor with just enough impulse  just above  the middle sensor  and can produce a tiny sound  , but that's rather an exception rather than the norm and in all cases not comparable to leggiero playing on an acoustic .

In non hybrid digital pianos the hammers and keys are linked together  with no degree of freedom  and cannot have independent stop positions. The only possible reasons  for  delta time of noise between the 2 mechanisms reaching their stop points would be mechanical tolerance and different compression of the felts ( hammer, key) . But also some DPs make more noise at key release because when the finger lifts the key accelerates on its own  and hits its upper stop, which is in some cases a thinner felt or even plastic contact. The later may bother as well because if may masks the subtle final release tail of the sound .

but the hammer is still linked to the key in a digital piano at all times .There no such thing as hammer free flight in digital pianos . Only hybrids are capable of that because they implement real acoustic actions . In traditional DP there is no need for escapement as the sensors performs that fonction . The hammers are just used for inertia and weight .  Most DP vendors are trying ( not always successfully) to minimise the key noise for the simple reason that key noise and hammer noise are already captured in the samples . Even Roland with its modelling technology in digital pianos have their own virtual key and hammer noise and hamner noise .

I know what you mean, but in this absolutism I disagree. The hammer can fly around the distance of the red arrow, WHEN the key is abruptly stopped before the note-on contact. You can simulate this by blocking the key with something, I used some cardboard on the following photo. Moving the key slowly to the cardboard note-on is not triggered (the second key contact is not reached). The same with key moved quickly and the note is triggered. The hammer has been "thrown".

This is not a typical key movement of course, agreed! ;-)

I think that perceived latency should be chosen, that works best with the owners Digital Piano (which is not a wooden Grand Piano).

In a DP roughly two times the energy is landing in the key noise (key stopper + hammer stopper, synchronous). In a GP roughly only one half lands in the key stopper and the other half in the vibrating strings and hammer noise.

Together with lower volume levels often used with DPs, it is another situation - on the timeline, on the volume levels and on the sound spectrum of the key noise. Eventually an additional workaround could be the individual reduction of Pianoteq's virtual hammer noise when playing at lower volume levels.

I'm thinking of it more as an optimisation problem. Latency too high feels sluggish, horrible in my opinion, yes. Latency too low it feels wrong with the inertia of a hammer weighted keyboard and the key noise. IIRC some Kawai DPs have the option to choose just the first of three key contacts to trigger the note (to mimic a quick Hammond or Synth response). Would be interesting, how you like that modus together with a Piano sound and the smallest buffersizes your system allows.

That the buffersize has to be always enough to avoid underruns, is just a necessary technical prerequisite here I would say (personally I also need very low jitter, I'm running a self-compiled Realtime-Kernel with precise 1000 Hz systemtimer).