I’m getting like 10-20tps with some other similar-sized models, but didn’t benchmark qwen yet. Note that they are mixture-of-experts, so they are evaluating only a few billion parameters each time.

Please ignore the post if you are not interested.