All I tell 'im is to set that core o' his on the monument, and watch."
Cores "remember" the World that Was. They continually produce energy, and were once widely used as power generators in Caelondia.
They are not made of Windbags, but rather burstone infused with the essence of Windbags from Burstone Quarry. The Cores are like diamonds -- years and years got to go by until the rocks down at the Quarry turn into something that special.
"The Quarry came with a lifetime's supply of Windbags. Windbags smaller than the eye can see are nestin' in those rocks. Not only are those rocks a source of life... we found that the oldest ones 'remember' things. But there's even more to it than that. They're recordin' everything, all the time, takin' it all in. No wonder all those beasts were drawn to the Quarry: the rocks were callin'em." -- Rucks, Burstone Quarry
After the Calamity
"The Cores contain the memory of the land from before it was undone. The City Crest flows with their power."
- -- Rucks
Outside the Bastion, Cores mainly hold together patches of the World That Was. Therefore, if you remove a Core from its area, one had best be prepared to leave post-haste. Without a Core, the ruins fall apart -- most won't even come together unless there's a bit of the Core's power around.
Some places are just too tough to fall, however. The less influence humans had exerted over an area, the higher the chance it pulled through the Calamity. The Wild Outskirts, for instance, was so "raw" several patches remained intact. Others, like Mount Zand, had holes through and through.
There are only a limited number of Cores -- things like those just can't be mass-produced -- but there are numerous lesser stones known as Shards that contain a little bit of the same power.
The Cores didn't just put out, they took everything in. They "remember" the World That Was, and with that memory (and the help of the Bastion) Cores hold the ability to restore parts of the old world.
In relation to the fact that Cores retain memories, there are places post-Calamity that seem too fantastic -- like they couldn't exist just as they once did, or have existed the way they do now. An exaggerated "memory" of the place might explain that.