Drop the "Warning: excessive partial hash collisions detected" for salt-only and blob-binary formats

[magnumripper]

Due to how FMT_BLOB formats (office for one) are implemented, they can't have binary_hash functions. Either mute that warning for them (very easy) or fix the problem (very hard, I think. Can't remember the details but I think I've tried and gave up).

[solardiz]

I don't recall seeing this warning for such formats, but maybe I'm not using them enough. I think muting the warning is fine for now.

BTW, what about salt-only formats in general, not only FMT_BLOB? Since they have zero binary size, surely they'd also have "hash collisions" for the binaries when many are loaded at once, wouldn't they?

[magnumripper]

what about salt-only formats in general, not only FMT_BLOB? Since they have zero binary size, surely they'd also have "hash collisions" for the binaries when many are loaded at once, wouldn't they?

I haven't seen that happen, I presume that is muted already - or maybe this isn't about FMT_BLOB at all but all salt-only ones.

[magnumripper]

what about salt-only formats in general, not only FMT_BLOB?

Yes, I confirmed the issue affects salt-only formats in exactly the same way.

My plan is to simply bump the threshold by 10x (or 100x?) for these types of formats, so we'll still get warnings/disabling but only for higher numbers.

BTW I noticed that office-opencl use fmt_default_binary_hash and got the full warning "(...) the "format" lacks proper binary_hash() function definitions" while CPU format use fmt_default_binary_hash_[0-6] which mutes that second part of the warning message. This is a bug in the Office CPU format, I need to review all FMT_BLOB formats.

[magnumripper]

Either mute that warning for them (very easy) or fix the problem (very hard, I think. Can't remember the details but I think I've tried and gave up).

It's not hard at all btw, trivial fix. EDIT: Oh, I meant fixing so they can have a binary_hash function of some sort. That's hard, if even possible/sensible at all.

[magnumripper]

I see now FMT_HUGE_INPUT formats have the exact same problems - binary hash functions not supported.

[magnumripper]

Either mute that warning for them (very easy) or fix the problem (very hard, I think. Can't remember the details but I think I've tried and gave up).

It's not hard at all btw, trivial fix. EDIT: Oh, I meant fixing so they can have a binary_hash function of some sort. That's hard, if even possible/sensible at all.

Wow, it turns out it's not only possible and sensible, we even have it already in the WPAPSK formats. This changes my road map a little here.

[magnumripper]

I meant fixing so they can have a binary_hash function of some sort. That's hard, if even possible/sensible at all.

Wow, it turns out it's not only possible and sensible, we even have it already in the WPAPSK formats. This changes my road map a little here.

WPAPSK doesn't have any get_hash functions though. I guess the binary_hash ones can only speed up loading. I hope (and think) this is allowed/supported (or we'd need a self-test disallowing it).

Edit: I'm pretty sure we could add the corresponding get_hash functions. I need to look into that.

[magnumripper]

WPAPSK doesn't have any get_hash functions though. I guess the binary_hash ones can only speed up loading.

@solardiz isn't it the other way around though? get_hash for loading and binary_hash for comparing? Then the above setup is just weird and the binary_hash functions should be dropped unless I can implement get_hash functions.

EDIT: No, binary_hash is for loading input hashes, and get_hash for what comes out of crypt_all.

[magnumripper]

WPAPSK does its final work in cmp_all as opposed to crypt_all. I can't remember right now but that might be a necessary property of FMT_BLOB - otherwise I can't see the point of that. But in that case I guess we can't trivially implement get_hash functions.

[magnumripper]

Indeed the combo of having binary_hash but not get_hash is fully supported, and is reasonable for BLOB formats. I tweaked the OP warning, added a few trivial self-tests and fixed the bug in Office formats (implemented proper binary hash functions).