I see current proj is using cuda core for fma, and from profiling, single batch seems more likely to be memory bound op. So for the batch > 1, it would require more computation and tensor core maybe needed in that case?

Yeah that's correct, tensor cores are not useful for single-user token generation, but they become critical for batch inference (they could be useful for prefill acceleration but that's not part of the workloads I'm personally interested in atm).