I concur that it isn't entirely the outside temperature. When it is colder you should probably start with more lit charcoal. Maybe your basket isn't allowing the proper airflow to the charcoal. Damp charcoal maybe? At that temp rain may be going in through your exhausts and hitting your charcoal. That is one advantage of the single 2" exhaust with a 90, keeps out a light rain.
XL Big Green Egg, Weber Performer Platinum, WSM 22.5"