BMW did a concept car a few years ago with a fabric skin, and apparently it's a regular part of their design exercises.
The biggest issue is damage and washing. Perhaps cheap to repair, but difficult and even what would normally cause a stone chip could cause a considerable tear. And how on earth do you wash it?