There are a bunch of options - it's too much to go into deeper detail, but there are a number of resellers for pay-as-you-go SIM cards.
There's an overview of tarriffs (although only in German) at
http://www.teltarif.de/mobilfunk/datenrechner.html that may give you an idea what to expect.
Generally speaking, when you're only staying in the country for a few days, it is best to pick up a starter pack from the local shops (Kaufland has some of them near the checkout) and just use whatever came preloaded with the card.
As far as network operators go, Telekom and Vodafone are usually at the upper end of the price range while O2 delivers the best bang for the buck.. and e-plus should generally be avoided.
If you want to reuse the card at a later stage, pay attention to the details - some cards expire if they haven't been topped up for six months, especially on the O2 and E-Plus networks.
You will also have to activate the card with proof of residency, either in the shop or online. They usually take the hotel address just fine. (Network operators don't really care for your address, but it's a legal requirement to give one, in case you want to use the SIM for a pipe bomb or something...
)