Brewer, an independent member of Cornwall Council, made the comment to a Disability Cornwall member at a stall at County Hall in Truro in 2011.
The Standards Committee demanded Brewer make a formal written apology to the charity.
“While I meant no offence by my remarks to you I can see, in retrospect, that they were ill-judged and insensitive and should not have been made at all,” he wrote.
In an interview with BBC News, Steve Paget, the chair of Disability Corwall said, “It is of great concern to us that anyone with such beliefs could be a representative and elected member of our local authority.”
“We did expect no less than this councillor’s resignation be called for,” he added.
Brewer has refused to resign, though he has been removed from the Independent group.
“I meant no offense,” he said about the comment. “I would defend disabled children to the last.”