I don't want to discourage you but if you ask me, I think referees need to have a higher level of fitness as the football players themselves. A football team consists of 11 players. Those 11 players have their own part of the field where they are responsible for. When a strikers has the ball, the defenders have a bit of time to rest.
A referee has to be on every spot of the field, as close as possible to see the faults players make. He has to be in the area of action, all the time.
I think you will benefit from cardio workouts and interval training. Try to focus on legs more than on shoulder routines or bicep curls if you want to do some strength training.