Creating a New Rules Set
So you know you want to run a game in a specific setting, and that setting comes with a special form of abilities. Many settings of various genres have their own unique powers. This can be anything from a different kind of magic, like alchemical transmutations or Ninjuitsu, to having a game in a high tech world with exo-suits, to humans being capable of amazing feats of near magical martial prowess. There are a lot of factors to consider when creating a homebrewed rules system.
Can I do this with what I already have?
Before you create your own homebrew consider this question carefully. In a Pathfinder game where you want to emulate a manna or chakra system think about if it might be easier to just use Psionics or spell points from 3.5 Dungeons and Dragons. If you want your World of Darkness vampire to be able to call down lightning storms consider just importing a Werewolf or Mage power that does the same thing. Look at the mechanics of your game carefully, consider what you could do by just reflavouring things a little, or a lot, and changing one or two basic mechanics. This also will require you to think about how precisely you want to be able to emulate the source material. Sometimes instead of a new system all you really need is a new coat of paint on an existing system. But for those other times there is a lot to consider.
How does this interact with other mechanics?
If you are adding a new option or feature to a system without removing old ones think about how they work together. Can a wizard dispel a Fullmetal Alchemist style transmutation? Does a Gellar field protect a ship from Protoss psionics? Can a Decker hack into a Gundam and if so what can they control remotely? If you've made your homebrew system correctly most of these questions will have straightforward answers. If you can't explain how 2 systems interact in one or two sentences you should consider simplifying it.
How do I balance this?
The importance of this question varies a lot. In a very mechanics driven game like Pathfinder or GURPS balance is more important than it is in something story driven like World of Darkness or Shadowrun. But in all of these games something unbalanced can become a clear problem. If one party member can bend water, and another is proficient with a sword and a third can bend all elements; is this game going to work? If it isn't can you make it work by tweaking the mechanics? This is especially important when a new system is introduced akin to a magic system, but the mundane characters don't get any new options for dealing or competing with it.
Is it simple enough?
This is pretty pivotal. Even if you make a perfect system that is 100% faithful to the original source material it's no good if you're the only one who can understand it. Ideally this system should be explainable to someone who has never experienced the original material. As a warning, it can be deceptive to compare your system with existing ones. Saying "My homebrew is no more complicated than Mage" does not automatically mean it's simple enough. People put a lot of time and effort into learning some core systems because they know they'll use them again and again. Furthermore players have help understanding from forums and being able to reread the rules and look things up. With your homebrew all someone has is what you give them and the only person who can clarify is you.
Is it working?
None of the other things matter as much as this. Is your homebrew enhancing your game or making it worse? This can include a number of things. Consider if the system is still fun for people not using what you created. Think about if you have to be having fun as the homebrewer seeing your creations come to life. Along with this idea is the question of if your homebrew achieved the desired effect. Does it make your game feel like the source material? Does it help the players think and act like the characters they're trying to emulate? If so you've done a great job.
Is it perfect?
No, it's not. It won't be perfect the first time you play it, and it won't be perfect the last time you play it. Every system has it's quirks and problems and oddities and that's okay. Nobody is perfect and no gaming system is perfect either, the important thing is a willingness to evaluate and change. Let's say you developed an amazing and clever system for perfectly replicating the classic FPS Doom as a tabletop game. First of all, wow, super neat idea. But second of all you may find your game suffers a bit when every encounter starts off with four players each saying "I fire my BFG until it goes click. Make a looting for ammo check and then repeat". Designing a fair and balanced system is hard, but breaking one is easy. It doesn't take long for players to find the most efficient approach and if you give them one option objectively better than the others they are more likely to use it all the time. There's no shame in admitting that you have to adjust things, and make sure you tell players why something is getting changed, they need to understand that a homebrewed piece of content is a work in progress and sometimes it needs tweaking.
Homebrewing anything is an exciting and sometimes daunting endeavour but it's something I heartily recommend to anyone. Even if you fail you're guaranteed to learn a lot about the game you're playing, the people you're playing with, and yourself.