If you don't keep up with the mini-game, then the attack can backfire. Do the best you can to get the best results for your Secret Attack.  If you are having trouble with a particular mini-game, you can either work on your skills in Practice or else try to use abilites that use a different type of mini-game.