Although your total testosterone is lower than what most would call normal for a man your age it's impossible to say if you have low levels without a Free Testosterone reading. Free Testosterone represents your active and usable testosterone. The reading you posted of 560 is your Total, this represents all the testosterone in your body but most of your Total testosterone isn't useable. You need to find out what your Free Testosterone reading is.

Do you have any other symptoms other than depression, low energy and apathy?
Did you ever get your adrenal and thyroid issues under control?

Your doctor is correct, if possible it would be best to get these things under control without meds if possible, but you'll need to answer the above questions if that's going to be possible.