Yes, I was going to say that I've used double stick tape before also.
And I've heard of people sticking their stamp to an old cassette tape for a very temporary mounting. You can also stick it to a child's block toy, or anything flat, but like buggainok said, use a foam stamp mat underneath to give cushion.
Make sure when you trim the stamps, cut close to the edge with your scissors straight up and down, or you'll undercut the stamp, which causes a crummy stamped image.
The best "permanant" ways of mounting your stamp, to be also removable for easier storage, are the EZ Mount system, (like Understandblue said above), and the Halos system of Velcro mounting.
You can find Halos here:
HALOS Rubber Stamps HALOS 1 System
Hope that helps!!