Skip to content
DataSpaceDataSpace

Local AI

In order to enable local AI capabilities an additional server is required with the following specifications:

Minimum (CPU Only):

Section titled “Minimum (CPU Only):”
  • CPU: 64 cores
  • RAM: 256GB
  • Storage: 100GB SSD

Production (GPU):

Section titled “Production (GPU):”
  • CPU: 64 cores
  • RAM: 128GB
  • GPU: 1x H100 80GB
  • Storage: 500GB NVMe SSD

We are leveraging a 120 Billion parameter open source model and therefore greatly benefit from GPU acceleration. While the system is flexible enough to handle any open source model, the smaller models don’t perform as well and often struggle with tool calling.

To use such a server, run Ollama on it and set OLLAMA_CLOUD_BASE_URL to its address. If it needs no token, leave secrets/ollama_cloud_token empty. See Configuration.