ADMG's suggestion is sound, but I'd recommend a slight variation: instead of labelling the books with their acronym, label them with a broader category. For example, the Monster Manual already has "Monster Manual" on its spine so there isn't any need to put MM on it, though it would be good to place it with other monster books (like Fiend Folio).
The way to sort them depends on how you use them. How open are you to using 3rd party material? If you are very open to it, then it is better to place them with their nearest 1st party equivalent, while if you are less open to it then the books should be sorted by publisher. Likewise, how often do you port Pathfinder material into 3.5 and vice-versa, and how often you use setting material outside of its setting. If your goal is more to serve your borrowers, then instead substitute their preferences for your own: for example, if you use a variety of 3rd party material but your borrowers almost always stick to 1st party or a narrow group of publishers, then it is best to divide it by publisher, or if you use Eberron material only in Eberron but your borrowers all use homebrewed settings it would make more sense to place Races of Eberron beside the other Races books rather than beside the other Eberron books.