IIRC, ability damage is temporary, ability drain is permanent. Restoration is a 4th-level cleric spell that cures the permanent drain, so it's not much of a sidequest.
Lamia (MM, p165) drains wisdom. Never fought one.
Stirge (MM, p236) hits Con. One stirge is no problem, but they usually come in swarms. Though one area-effect spell, and a group is no problem either.
There are also monsters with poison that do ability damage/drain, but the PCs get a saving throw.