You have found the ongoing problem with thin-walled offsets. The same scenario why we insulate our household thru the attic... loss of thermal energy.
Check and seal leaks... and insulate via natural fiber (non-synthetic) blankets by draping over the smoking chamber.
Thicker steel (mass) helps with heat retention, too.
Custom Klose Backyard Chef... of Snot, and a 22.5 WSM
Lang 84 Deluxe kitchen, Lang 60 Mobile - The Damsel II, Lang 48 Patio - The Damsel, Bubba Keg Grill - RIP, Double Barrel Smoker and a BSKD