My drum never has had any wild temp fluctuations. Mine is built identical to yours except I have a 6" x 2" threaded pipe for exhaust. I think if you add the exhaust it will probably draft better. The only other reason to have temp swings is air leakage somewhere. Be sure not to leave the lid off for any length of time or you will have a temp spike. If I have to open the lid I get in and out quick.
Weber 22.5 WSM (2) UDS, Chargriller Acorn, Pit Barrel Cooker, Pitmaker Vault, Safe and Sniper on Trailer