Eggs have a lot of thermal mass so they take a longer time to drop in temperature. That being said, you may have a leaky or even a bad gasket. You can do the dollar test to find out.
Also, my cast iron damper takes a lot longer than the ceramic cap to snuff out the fire.
Jim - Another transplanted Texan
Former KCBS CBJ
Large and Medium Big Green Eggs , Black 18.5" WSM, Blue Weber Performer - Stainless, Green Weber OTG Kettle , Brinkmann SnP Pro, and a Stainless UDS. One retired Portable Kitchen grill.
Red Thermapen, Maverick ET-732, EdgePro Apex Sharpener.
Avatar is the original 1951 Weber Kettle