I do this quite often nowadays. It sounds silly, but I guess it is far worse. Read on.
I tend to refrain from going out, watching TV, or enjoying in general.
I feel I could do something 'worthwhile' in that time, instead of wasting in some frivolities.
That's good, right? but after refraining from fun, I go ahead and do nothing of value in that time, while everyone else is enjoying.
The pressure to do something meaningful drives me to avoid certain time-wasting activities, but I replace those activities with something equally useless.
And I end up miserable because I didn't do anything worthwhile, nor did I have any fun.
I guess this "I am going to do something about it" feeling is the biggest culprit. I have no intention of doing the task, yet I force myself to do something towards it (keeping some time aside for it, in this case). That, I guess, is reason enough to let the mind wander, instead of actually doing something.
If I said "I don't have any interest in this thing", then maybe I can pursue something I really want to do instead.
I guess I was just fooling myself into thinking that I am someone who does all these impressive things. Maybe by being truthful, I can bring myself to achieve some of them.