Riding with panniers is much nicer than having a load on your sweaty, gross back.
How far are you thinking you might bike? If you want to go 500 miles once a year, it might be worth it to just get a road bike and add some touring gear when you go, but if you're going on centuries to go camping every weekend, you might want a touring bike.