Someone who is unaware of your presence is just flat-footed, not helpless.
For someone to be helpless, they have to be unconscious (including sleeping) or paralyzed. There are a few other causes of helplessness that come up, but they're not very common. If you want to coup de grace someone, you're probably going to need to inflict one of those conditions on your target first. Some examples:
- Sleep and Deep Slumber spells make one or more targets fall asleep (although they have terrible HD limits).
- The Hold Person and Hold Monster spells cause paralysis.
- There's a Shadow Hand maneuver (don't remember the name unfortunately, and I think it's somewhere between 5th and 8th level) that lets you paralyze a flat-footed target for a few rounds.
- If you drop your target's ability scores to 0, they are paralyzed (Strength, Dexterity), unconscious (Intelligence, Wisdom, Charisma), or just die (Constitution).
Separately, I seem to remember discussion about some rules in Book of Vile Darkness about using Profession (Executioner) to coup de grace someone pinned in a grapple. You might want to look into that.