Fix generate() when tokenizer is unset and add regression tests

[DityaChawla]

Summary

Fixes tokenizer-free HookedTransformer.generate() paths where a tokenizer is not required.

Fixes #483.

Changes

• removed the unconditional tokenizer assertion in generate()
• updated get_attention_mask to accept tokenizer=None
• added regression tests for tokenizer-free generation

Validation

• verified tokenizer-free generation works with stop_at_eos=False
• verified tokenizer-free generation works with explicit eos_token_id
• added regression coverage in tests/unit/test_generate_no_tokenizer.py

[jlarson4]

Hi @DityaChawla! Thanks for taking on this change. A couple things before we can get this merged:

• This is based on an older version of TransformerLens, could you please pull down the latest version of dev and apply this change there? (get_attention_mask now lives at transformer_lens/utilities/tokenize_utils.py around line 229)
• Currently the function body does tokens.ne(tokenizer.pad_token_id) which crashes if tokenizer is None. It already has a if tokenizer is None: return attention_mask early return at line 249-250 (all-ones mask) — but that's gated on an implicit check that only fires when tokenizer is truthy.
• Add a test with use_past_kv_cache=True explicit and another with use_past_kv_cache=False to prove both paths work
• generate() calls several utilities that may assume a tokenizer. Worth checking for self.tokenizer. in the generate method and verifying each is guarded.