Without seeing this boat,
I would be hesitant to make any definitive remarks. I would say, however, that the amount of "sag" you mentioned would appear to be pretty serious to me. If the mast is "sagging" that much, it also means that the standing rigging could be placing an unbalanced load on the hull/deck joints unless it has been slacked off. All in all, not a good situation. I personally believe that rebuilding the mast step to seat the mast where it's supposed to be could be a major job and a potential deal breaker unless the price of the boat has been adjusted to reflect this condition -- and you are willing to take this job on, either yourself or to have aa professional do it. I would also be concerned with your surveyor's remarks stating this would not be a "big job to correct." In any event, if you continue to be interested in this boat, I would (1) get another opinion from a qualified surveyor (not a full survey, but just an opinion on this particular problem) and (2) ask a professional repair shop for an opinion and/or estimate on what it would take to repair this problem.Again, not having seen this boat/problem, it may be that it really is not that big a deal. But if it is, you want to know about it before it becomes your problem.