Since Mordechai did not rise for Haman, even out of regular respect, Haman’s seething anger was directed specifically at him.
Rav Chaim Kanievsky (shlit”a) writes that Haman was shocked that Mordechai was not groveling to him, begging for his life.
The Vilna Gaon and Yosef Lekach write that Haman had suspected that Mordechai had previously not bowed to him because he seemed to be friendly with Esther. Now that Haman was invited to the party and Mordechai was not, he considered his suspicion ungrounded. Therefore, Mordechai’s refusal to even stand for him was doubly upsetting.
The Targum explains that another reason Haman became so upset with Mordechai because Mordechai was sitting on the ground, looking at his shoe. The reason for this is that the document of Haman’s being Mordechai’s slave was written there.
According to Ginzei HaMelech, quoting his father, R’ Efraim Mordechai Ginzburg, Mordechai chose this moment to look at his shoe to strengthen his resolve, reaffirming his realization that H-Shem will always somehow make everything work out for the best in the end.