As far as I know, you can't in a strict way (unless using some kind of virtual machine or using cgroups which is not so easy; you can see this answer from @muru.).
You can limit the memory available with ulimit, but this will simply have the effect of telling your program that there is no more memory when doing an allocation, or crashing it with a signal if it doesn't handle the out-of-memory condition. Look:
zcat /var/log/syslog.2.gz
it works, lot of output
ulimit -d 100
ulimit -m 100
(This is limiting the memory for data and for core to 100 kB)
[romano:~] 2 % zcat /var/log/syslog.2.gz
/bin/zcat: xmalloc: .././subst.c:3542: cannot allocate 267 bytes (53248 bytes allocated)
But the shell is still able to see all the memory:
[romano:~] 2 % free
total used free shared buffers cached
Mem: 15340736 5148596 10192140 368776 284192 2794848
-/+ buffers/cache: 2069556 13271180
Swap: 31999996 0 31999996
Edit: more info in this unix.se post.