AI Providers or Local Inference
Is it possible to provide a list of AI Providers OR allow the user to add one?
- Enterprise Copilot (we need this)
- Github Copilot
Can you allow easy selection of an Offline Model for local inference?
It would be nice to use an offline model and harness a Discrete Graphics card or an NPU for things like Code Instruction\Completion\Questions. Of course, I understand that means knowing quite a bit about the model and the memory available to it, but that seems reasonable to expect of those who choose to try using it.
I am asking about the offline model largely out of concern for the Environment. Although these cloud services can offer useful suggestions about the code I am writing, they consume significant amounts of water and power. Further abuse of the environment at this moment seems arrogant, even ignorant.
Lastly, I anticipate a steep increase in the cost of cloud LLM usage, and I don't want to eventually end up investing too much effort in a service that has to somehow become profitable, because I will have to simply stop using it.