AI: Create a domain-specific LLM

Question

Apologies in advance for a very broad question. I have a friend who works as a grant writer and she has a corpus of successful grant proposals. Is there a way to easily create a domain-specific LLM that trains off of these proposals for the creation of future proposals?

Answer 1

That depends on your definition of "easy". Google "llm domain-specific training". There are services out there that can help you, or if you are extremely masochistic, you can find pointers on downloading a small LLM and doing your own specific training.

The specific answer to your question is "yes, for certain values of 'easily'."

Answer 2

There are open source LLM models which you can fine tune and transfer learn. I think transfer learning is what you need to look into. So you are starting from a pre-trained LLM. However, "easily" is somewhat dependent on where you are starting from. Even starting with a pre-trained LLM you will need quite a bit of GPU in order to transfer train. As a minimum you will need something like a 24Gb NVidia GPU. You may need to rent cloud GPU.

You should learn what transfer learning means and how you would select a suitable pre-trained LLM. You might want to select one that has a selection of sizes (number of parameters) so you can start smaller to test your ideas and then scale up when you are seeing the kind of results you need.

I recommend you do at least the 1st fastai course if you are new to this. That will give you pretty much everything you need to make a decent run at this project.