It "drives him crazy" that people say 'at the end of the day.'
(Now why did I use double and then revert to single quotation marks? Because I went from quoting a particular person to mentioning a phrase in widespread use, but not quoting any particular person. There is no need for you to be so punctilious. Just don't call me inconsistent.)