There's nothing to feel guilty about in being idle. It's not reckless. It's an investment. There is nourishment in pursuits that have no purpose - that is their purpose.
Leisure is also a reward for the work we do. When we think about the ideal "Renaissance man," we see someone who is active and busy, yes, but also fulfilled and balanced. Getting to know yourself is the luxury of the success you've had. Finding fulfillment and joy in the pursuit of higher things, you've earned it. It's there for you, take it.
Make the time. Build the discipline.
You deserve it. You need it.
Your stillness depends on it.