Step 1: Know all the rules.
I'm not sure if there is a specific method to it apart from reading all the rules like a Lawful Evil Deity of Devils' Advocates. And since most rule-systems are pretty huge, you'll want to start in an area of the rules, that you want to play in a lot.
For example, I am most interested in (spontaneous) magic, so I'll read up on all the rules concerning this area, before I even go into other things: e.g. I know next to nothing about sneak attacks and little about melee combat (in 5E).
Make sure to read all the guides, that others have written, on the relevant subjects. If you use computer/online resources (like a rules wiki), make a bookmark folder just for that purpose, make bookmarks on all the important rules stuff (e.g. conditions, combat calculations, lighting/environment effects, certain feats, ...) and organize those bookmarks if necessary.
For example, I have a bookmark folder full of interesting (sorcerer) spells, arranged according to necessary components (V only, S only, S+M only), a huge exclamation mark for all concentration spells, and base info about the spell in the bookmark name: e.g.
[0] Mind Sliver [Ench; INT save] in the "V only" folder, letting me know it's a 0-level enchantment spell (i.e. cantrip) against an INT save, that I can cast without somatic components. That way I always have a quick overview of options that go along with a Silence spell (S only or S+M only), with the War Caster feat (S+M only) and while my hands are tied (V only).
You can also build a strong character around utilizing a specific ability or spell. Then you will look for ways to enhance or broaden its application. And then make sure you subject yourself to the hive mind, i.e. you post your build, strategy, idea along with a thematic picture to capture the attention of others on a relevant forum and ask for advice/criticism.
For example, the user "bendking" over on the 5E area in the GitP (Order of the Stick) forum posted
his "God of Lightning" build, after which others (i.e.: me!
) engaged in a lively discussion on how to improve it.