World Travelers, Transmigrator, Other Wolder or "Foreigner"「異世界人, Isekaijin literally meaning "Other World People"」, refers to people who came from another world through sheer coincidence and are called Strays (偶発的にやって来る異世界人 Gūhatsu-teki ni yattekuru i sekai-jin, lit. "Otherworlder transported by accident") or through reincarnation and are called Reincarnated (転生者 tenseisha) or through summoning and are called Summoned (召喚者 shōkansha).
In history, there were several cases whereby some people were either reincarnated, summoned, or came to this world by accident. In some cases, they were summoned here, hence the Summoned Ones. On the other hand, they could have arrived from another world by accident or were simply reincarnated. There was not a fixed way of how the World Travelers came.
Being reincarnated tends to happen randomly, especially those whose souls with strong intentions engraved upon them. Although there are those among the reincarnated who still remember their past life, to have reincarnated from a different world is exceedingly rare. Moreover, one would normally be reincarnated as a human. However, there are very few exceptional cases. For someone to be reincarnated a monster, and from magical energy itself, is something that remains unheard of.
As for those who came to this plane by accident, it also happens randomly. However, unlike those who undergo reincarnation and have to be reborn, they arrive here with their original bodies intact and whole. In addition, they still retain their original memories.
In any case, setting aside whether method they came by, it is a known fact that those who came "naturally" will possess the special intellect and acquire some unique skills, based on the characteristics of the individual themselves, naturally upon their arrival.
It's pretty common that Summoned Ones tend to have a Curse of Obedience etched on their Soul to make them unable to go against the Summoner.
Existing records of reincarnated being renowned for their wisdom do remain, although there are most likely those that were never recognized.