I think it is important if you want to really control temp. Leakage is what everyone tries to avoid since you lose control of the airflow. You could, and I did start out with just tapeing the fan and damper holes, but I can get much more precise control of the airflow this way.
You could tape the inside of the 2 other holes. Probably need to clean the cooker in that area pretty well to get the tape to stick, but the fan mounts tighter w/o the damper. Saying that, if my damper wasn't mounted with a bolt instead of a rivet, I'd have done something like you suggest.