18 Commits

Author	SHA1	Message	Date
v0xie	fe1967a4c4	skip multihead attn for now	2023-11-03 17:52:55 -07:00
v0xie	d727ddfccd	no idea what i'm doing, trying to support both type of OFT, kblueleaf diag_oft has MultiheadAttn which kohya's doesn't?, attempt create new module based off network_lora.py, errors about tensor dim mismatch	2023-11-02 00:13:11 -07:00
v0xie	a2fad6ee05	test implementation based on kohaku diag-oft implementation	2023-11-01 22:34:27 -07:00
v0xie	6523edb8a4	style: conform style	2023-10-22 09:31:15 -07:00
v0xie	3b8515d2c9	fix: multiplier applied twice in finalize_updown	2023-10-22 09:27:48 -07:00
v0xie	4a50c9638c	refactor: remove used OFT functions	2023-10-22 08:54:24 -07:00
v0xie	de8ee92ed8	fix: use merge_weight to cache value	2023-10-21 17:37:17 -07:00
v0xie	76f5abdbdb	style: cleanup oft	2023-10-21 16:07:45 -07:00
v0xie	fce86ab7d7	fix: support multiplier, no forward pass hook	2023-10-21 16:03:54 -07:00
v0xie	7683547728	fix: return orig weights during updown, merge weights before forward	2023-10-21 14:42:24 -07:00
v0xie	2d8c894b27	refactor: use forward hook instead of custom forward	2023-10-21 13:43:31 -07:00
v0xie	0550659ce6	style: fix ambiguous variable name	2023-10-19 13:13:02 -07:00
v0xie	d10c4db57e	style: formatting	2023-10-19 12:52:14 -07:00
v0xie	321680ccd0	refactor: fix constraint, re-use get_weight	2023-10-19 12:41:17 -07:00
v0xie	eb01d7f0e0	faster by calculating R in updown and using cached R in forward	2023-10-18 04:56:53 -07:00
v0xie	853e21d98e	faster by using cached R in forward	2023-10-18 04:27:44 -07:00
v0xie	1c6efdbba7	inference working but SLOW	2023-10-18 04:16:01 -07:00
v0xie	ec718f76b5	wip incorrect OFT implementation	2023-10-17 23:35:50 -07:00