mrseeker
patreon
OPT-30B-Erebus-V1 (pipelined)
🕑 Added 2023-01-09 11:37:44 +0000 UTCIt took me a while, but I am currently training Erebus-v1 op OPT-30B. I wish I could get higher, but this is a good start to seeing how my training schedule behaves on bigger batches.
I have to say, I was shocked at seeing how much this would cost. To give you an idea, the estimate is between 2500 to 3000 USD for Erebus-v1 (451,398 training samples) and will take around five days on 5x A100. Knowing that Erebus-v2 is 970,832, training samples would cost 5000 to 7500 USD to finetune OPT-30B-Erebus-v2.
I might consider doing OPT-66B when I have the funds, but at this time, I am seriously considering adding a new tier for my Patrons to fund this big lousy boy or even setting up a Kickstarter for this. The 1 USD tier is to give a little "tip" for my efforts. However, the current tips only account for 20% of my monthly payments.
Comments
Feudal Agitator
Definitely would be game for higher tiers if it helps. Forgive the noob question, but what's the practical effect of 30B over the 20B version?
