The Charger is fat and ugly. Its dimensions are all out of whack, like a misshapen hunchback monkey boy. The Challenger on the other hand has an attractive shape and fits 5 people comfortably, thereby making the Charger a complete waste of time and space.
Check out this artist impression of a two door Charger:
.. looks great! But elongate the cabin and it becomes this monstrosity:
Which to me looks like a Frankenstein mutant hybrid of a muscle car and full-sized family sedan - which is basically what it's trying to be anyway!
Why don't we make a hybrid of a Corvette and a mini-van to make everyone happy! Not. Don't mix and match.