There is always a certain stress to travel. There are so many systems to keep track of: the system that gets you to the airport, the system that allows you to park your car, the system that gets you through security, the system that gets you on the plane, the system that gets your luggage loaded, the system that gets your biological needs attended to in the air, and then all that in reverse at your destination. Then comes the system that gets you lodging, the system that gets you transportation (car rental for newbies, ugh), the system that keeps you fed and entertained while you're gone. For the most part these can be mastered through experience, but I would hate to have to try to master them all from scratch. There is so much to know (much more than when I was young) and only so much that signs, warnings, and instructions can clue you in on. A young, pliable mind of average intelligence can probably handle it, but to an old or substandard intellect, less prone to quick and accurate observation and inference, it must be horrifying, especially when so much of the process is filled with strident commands from punitive authorities and dire warnings about the failures to comply quickly. And let's not forget the ever present impatience of the skilled travellers you might be holding up. I have friends who are knocking on the door of 60 and never travelled significantly. I cringe at the struggle they are facing when they decide to finally take that dream trip. Even if they manage to adapt to all the norms, the rush-and-wait rhythm is exhausting until you are used to it. Inexperience both in the planning and performing of vacationing will almost certainly swamp that dream with disappointment.
But even if you have all this down pat, like Yours Truly, there are unknowns. In part or whole, all these systems involve humans in some capacity and that introduces random variables. As a result, these systems can change subtly and without warning -- one plane leg might be enforcing the carry-on limit and another may not; one security line might tell you to remove your watch, and another may not; one hotel might let you check in early, another may not or try to milk you for money to do so; one tour guide may be brilliant, another full of shit.
Even more uncontrollable is your personal situation. Are you travelling in a pair, or a group? If so, then every decision on what to do is a negotiation. There are people who are very happy ordering room service and watching pay-per-view movies in their room in paradise. There are people who will fly thousands of miles and then go to a mall and eat at Red Lobster. There are people who will pick fights with everyone in the service industry and believe they are being cheated at every turn. There are people who will happily sit on the beach for twelve hours a day. There are people who will plan everything down to the minute. There are people who madly rush to see everything they possibly can for 30 seconds, as if they are bagging sites like coins in a video game. There are people who wander aimlessly and settle for the entertainment comes to them. There are people who are prompt on the dot, and people who linger and wallow in minutiae until you are late for everything. (In my experience nothing is more dangerous to enjoying a trip than an incompatible group of personalities.)
Maybe you should travel on your own, you say? It does have its benefits, not the least of which is doing what you want when you want without explaining yourself. That said, are you prepared to join, say, a catamaran snorkeling tour with nothing but families and couples and you on your own? How about asking for a table for one? You must realize that often, when you travel alone, the people you encounter regard you with a mild form of pity -- will that bother you? More importantly, can you live inside your own head, with your own thoughts, for extended periods if you need to? To many, this challenge is insurmountable.
What I'm saying is that vacationing, like life, is a complex activity, and needs practice to get right. Bad trips are learning experiences, both about the external forces and yourself. In time, it can become a great pleasure; your vacations can form some of your most treasured memories and can mark the phases of your life, but if you push it off and wait until you are older to take that one dream trip -- your sense of optimism will be tested, even on something as innocuous as a Caribbean cruise.