• 29 Posts
  • 22 Comments
Joined 1 year ago
cake
Cake day: June 12th, 2023

help-circle











  • Btw I know this is old and you may have already figured out your hardware and setup, but p40s and p100s go for super cheap on eBay.

    P40 is an amazing $/GB deal, only issue is the fp16 performance is abysmal so you’ll want to run either full fp32 models or use llama.cpp which is able to cast up to that size

    The p100 has less VRAM but really good fp16 performance which makes it ideal for exllamav2 usage. I picked up one of each recently, p40 was failed to deliver and p100 was delivered while I’m away, but once I have both on hand I’ll probably post a comparison to my 3090 for interests sake

    Also I run all my stuff on Linux (Ubuntu 22.04) with no issues




  • I don’t have a lot of experience with either at this time, I’ve used them here and there for programming questions but usually I stick to 7b models because I use them for code completion and I only find that useful if it completes the code before I do lol

    That said, I’ve had overall good answers from either whenever I’ve decided to pull them out, it feels like wizard coder should be better since it’s so much newer but overall it hasn’t been that different. Wish phind would release an update :(











  • I use text-generation-webui mostly. If you’re only using GGUF files (llama.cpp), koboldcpp is a really good option

    A lot of it is the automatic prompt formatting, there’s probably like 5-10 specific formats that are used, and using the right one for your model is very important to achieve optimal output. TheBloke usually lists the prompt format in his model card which is handy

    Rope and yarn refer to extending the default context of a model through hacky (but functional) methods and probably deserve their own write up















  • Yeah definitely need to still understand the open source limits, they’re getting pretty dam good at generating code but their comprehension isn’t quite there, I think the ideal is eventually having 2 models, one that determines the problem and what the solution would be, and another that generates the code, so that things like “fix this bug” or more vague questions like “how do I start writing this app” would be more successful