This is known as a fork bomb.
Does linux have any protection measure to such program which can run out of memory?
Not really. Each fork produces a new process, with its own virtual address space and memory usage. So each copy is relatively small. Eventually, you'll use up all the physical+swap memory on the system, and the out-of-memory (OOM) killer will start killing individual processes. But the fork bomb will still be creating processes just as fast (if not faster).
One way to prevent this occurring in the first place is to limit the number of user processes, using ulimit -u (assuming you're using Bash; other shells will have equivalents).