You have no reason to go pact of the blade if you're only going to level 3. You can't take Thirsting Blade until level 5 of warlock. The only benefit you'll get with 3 levels of warlock is a magic weapon on-demand, and it would seem based on your build that the optimal weapon would be a rapier, a weapon that a bard is already proficient in.
I would suggest either of the other pacts since they do a lot more for you than pact of the blade without its exclusive invocations. Pact of the tome would allow you to use Shillelagh with a quarterstaff for your build. Shillelagh learned via pact of the tome functions based on Charisma, reducing the number of attributes you depend on. Combined with Booming Blade and/or Green-Flame Blade, you'll have comparable melee attacks to attacking twice with Thirsting Blade, eventually surpassing that at level 11.
The assassin level 3 ability is harder to use than you think. It's only an automatic critical hit if the target is surprised. Just going first isn't enough to get the auto-critical.