zig fmt fails with unhelpful error message when a comment contains a tab

[ikskuh]

Formatting the following code with zig fmt fails:

//	this is broken!

fail.zig:1:1: error: Expected test, comptime, var decl, or container field, found 'Invalid'
// this is broken!

It does not matter where the comment with the tab is, the error message stays the same, pointing to filename.zig:1:1, so it's hard to find what causes this

[LakeByTheWoods]

This is being caused by #2915

At the top of Tokenizer.next there's

        if (self.pending_invalid_token) |token| {
            self.pending_invalid_token = null;
            return token;
        }

token's start and end get set to 0 by self.pending_invalid_token = null; so the resulting error print thinks the token is at the start of the buffer.

I have two potential solutions:

1. Explicit copy

        if (self.pending_invalid_token) |token| {
            const copy = token;
            self.pending_invalid_token = null;
            return copy;
        }

Which I don't think it looks very nice, but it is at least (hopefully) clear what the writer is intending.

2. defer setting null

        if (self.pending_invalid_token) |token| {
            defer self.pending_invalid_token = null;
            return token;
        }

Which I'm not sure if it's too clever to the point of being hard to understand or a cute potentially idiomatic way of setting an optional to null when using its value this way.

I've tested both, they both correctly point out the tab token as the problem (though as an aside, the error message doesn't make it clear why it's a problem, ie. tabs and spaces both look the same in the message).

(or there's the 3rd option of disarming the foot-gun and keeping the original code)

[LakeByTheWoods]

Looks like this isn't the only time this has happened, the json.zig tokenizer uses option 1.