 michaelh
|
243f821d93
|
feat: Add 4-bit loading + LoRA support for low VRAM optimization (#60)
* Add files via upload
* perf: optimize abliteration matrix op (#46)
* perf: optimize abliteration matrix op
* refactor: comments and var names correspond with arditi
* refactor: fix comments and improve var notation
* fix: accidental line change and improve comments
---------
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com>
* Fix line endings to LF
* Add hybrid approach for GPT-OSS compatibility
- Check for LoRA adapters before attempting LoRA abliteration
- Fall back to direct weight modification for nn.Parameter (GPT-OSS)
- Ensures compatibility across all model architectures
* Fix projector bug, update print statement, revert README
* Revert README changes to match upstream
* Fix import sorting for ruff
* Fix reload_model for evaluate_model, add type hints and validation
* Apply ruff formatting
* Replace load_in_4bit with quantization enum
* Fix precision loss: use FP32 refusal direction directly
* Move r assignment into non-LoRA path
* Fix linting: apply ruff formatting
* Add auto-merge for LoRA adapters on save/upload
* Fix linting: apply ruff formatting
* Implement CPU-based merge for 4-bit models with OOM fallback
* Remove use_lora flag (LoRA always on), add user prompt for 4-bit export
* Fix: PEFT target_modules expects module names without path prefix
* Fix linting: apply ruff formatting
* Add LoRA fallback and fix quantization_config handling
- Add try/except around LoRA initialization with fallback to direct weight modification
- Only pass quantization_config when not None (fixes gpt-oss loading)
- Use simple forward pass instead of generate() for model test (avoids chat template issues)
- Reset non-LoRA models by reloading in reload_model()
- Check self.use_lora before accessing LoRA adapters in abliterate()
* Add 8-bit quantization support via bitsandbytes
- Add BNB_8BIT option to QuantizationMethod enum
- Add --load-in-8bit CLI support (auto via pydantic-settings)
- Update documentation in config.py and config.default.toml
- Useful for mid-range VRAM (12-16 GB) as balance between memory and numeric stability
* Improve LoRA merge warning and fix linting
* Apply final ruff formatting
* Fix CI: apply ruff import sorting
* Use tiny model for CI efficiency
* Fix import sorting in test_lora.py
* Fix formatting in test_lora.py
* feat: Show merge warning for all models (requires high RAM)
* style: Apply ruff fixes
* Fix undefined Style import in main.py
* Fix(model): Support MoE/3D tensors and enforce dtype safety in abliterate
* Fix(ci): Format model.py with ruff
* Fix(main): Remove invalid style argument from prompt_select and unused import
* Fix logic errors, memory leak, and redundant merges in main.py
* Fix linting and formatting issues (isort, ruff)
* chore: Simplify .gitattributes as requested
* refactor: Remove defensive try-except around LoRA initialization
* chore: Update uv.lock with peft and bitsandbytes
* chore: Regenerate uv.lock to include missing peft dependency
* style: Fix import sorting (isort) for CI compliance
* style: Simplify .gitattributes to single line as requested
* Address PR #60 feedback: Remove caching, fix LoRA reload, global LoRA usage, style fixes
* Address PR review comments: clarify code, fix quantization, rename method
- Add explanatory comments for warning suppression and gc behavior
- Remove redundant gc.collect() calls (empty_cache handles it)
- Fix output message order (ask merge strategy before 'Uploading...')
- Add comment explaining 8-bit quantization doesn't need compute_dtype
- Remove extra newline after dtype comment
- Add future-proofing note for hybrid layer support (#43)
- Remove leftover comment in get_merged_model
- Delete test_lora.py (debug script, not a real test)
- Add comment explaining needs_reload flag purpose
- Extract quantization config into _get_quantization_config() helper
- Rename reload_model() to reset_model_for_trial() for clarity
- Fix reload_model to respect quantization config (fixes evaluate_model bug)
- Remove unused gc import
* Restore gc.collect() before empty_cache() for large models
* refactor: Remove LoRA fallback remnants, simplify code
- Remove use_lora flag (always true since LoRA is always applied)
- Remove isinstance(PeftModel) check in get_merged_model() (always true)
- Simplify reset_model_for_trial() by removing defensive try/except
- Remove redundant gc.collect() calls (empty_cache handles GC)
- Remove unused gc import from main.py
* Address p-e-w review feedback: rename reset_model, remove loaded_model_name, fix type hints, remove GPT-OSS MoE, update assertion
* Restore skip logic for non-LoRA modules and fix 4-bit base_layer.weight access
* Remove defensive lora_A check per review - get_layer_modules already filters
* Fix try_add: nest component init inside Module check, add assert for unexpected types
* Add note about module.weight assumption for type checking
* Change 'Reloading model' to 'Resetting model' in logging
---------
Co-authored-by: accemlcc <accemlcc@users.noreply.github.com>
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com>
Co-authored-by: Hager <Michael.Hager@bruker.com>
|
2025-12-14 20:19:09 +05:30 |
|