I think he means by saying, "I only worry about sending the fruits of the garden which I cultivate", that we can only cultivate the gifts and passions we are naturally endowed with, and to focus on the process of creating. Not to get distracted by what you can't control, outside of your garden.