It was indeed the springs, I ordered some Acuity ones from Tegiwa, they are not much stiffer than OEM.
I saw nothing wrong with the shifter mechanism or the original springs that were in there, but as soon as I changed them the slop disappeared, now my shifter feels amazing, especially paired...