A similar matter came up in a recent question regarding two-handed weapons and armor spikes.
I'm perfectly ok with using the full-attack with the two-handed weapon and armor spikes as an "off-hand" attack (applying TWF penalties, of course), because the spikes are attached to the armor and they function exactly like, say, a monks unarmed attack (which can be made with whichever body part is available for it).
On one hand (no pun intended), as per RAW, the claw attack shouldn't be allowed, because you're using a manufactured weapon with your hands.
On the other, there's that nasty/handy rule that states that removing a hand from a two-handed weapon, and putting it back, is a free action (great for caster-fighters like the Duskblade). Just remember that you can't perform free actions when it's not your turn, so, if you don't put your grip back on the two-handed weapon, it won't be ready for AoOs.
Since it's homebrew stuff, you are, of course, entitled to come up with everything you want, so if the claw is to be used along with the full-attack of the greatsword, i suggest that you, at least, apply a TWF penalty to it.
If it's a question of flavor, you could instead allow the attacks to be done interchangeably.
Example: BAB 15/10/5
Attacks: any combination using claws and the greatsword (GS) with the normal attack bonus.
CLAW/GS/CLAW, CLAW/CLAW/GS, GS/GS/GS, GS/GS/CLAW, etc.
BTW, i'm not sure if i understood your thoughts correctly (it's late here and my brain is half asleep), but AoOs are already separate from the normal attacks...
Example:
If you're wielding a guisarme (two-handed polearm, 10ft. reach) and your armor is equipped with armor spikes, you can perform a full-attack with the guisarme during your turn and later perform AoOs normally using your armor spikes or the guisarm (whichever threatens the opponent) without penalties.