Thanks Joshua!
Looked at all your suggestions but no luck.
Output caching is turned OFF. Partial caching is turned OFF. Web Analytics are turned OFF. Files are stored in both file system and database. The Allocated Memory keeps growing, starts at 150MB and up to 550MB+. Process Physical Memory 380MB, Process Virtual Memory stays at 9GB.
To me it looks like i'm doing nothing strange. What would be an average memory usage for a super simple 3 page (home/blog/contact) site containing simple text and photos, nothing fancy?
I'm the type that first start to read the manual before playing around :) I can't stand it that I missed this part...