Mystic is a fan coined term that describes the power up that Gohan underwent during the Majin Buu Saga at the hands of the Elder Kai. This power up is said to unlock an individual's sleeping power and subsequently, taking them beyond their natural limits. In Gohan's case, this increase was so huge, that it even surpassed his father's Super Saiyan 3. In order to unlock the sleeping powers, the individual who's power will be drawn forth, must first subject himself to a ritual.
 Unlocking ritual
To get started, the Elder Kai first performs a weird looking dance that lasts for several hours, followed by the power-up session, which lasts several times longer even. The waiting was worth it though, as Gohan was easily able to brutalize the fearsome villain Majin Buu in physical hand-to-hand combat. The only way for Buu to repel Gohan's attack was by absorbing the fused character Gotenks.
This form appears two times overall in Dragonball Z. Once in the original manga, when Gohan challenges Majin Buu, and Gohan also uses this state to fight the monster Hirudegarn in the thirteenth Dragonball Z movie: Wrath of the Dragon.