I believe you are correct; if the unsafe code can cause undefined behavior if input data is not following a specific contract, then the entire function should be labeled unsafe so the caller knows that.

The other option is to check to make sure the contract is valid, and return an error or panic if it is not. That function would be sound, as no inputs cause undefined behavior.