10.3 Groq API
If you would like to experiment with publicly available LLMs but don’t have a sufficiently fast computer, Groq (https://wow.groq.com) is a very helpful alternative. This company currently offers developers a free option to run a few selected language models (llama, mixtral, gemma, whisper).
What is remarkable about Groq is the incredible speed at which prompts are answered—much faster than with good proprietary hardware but also significantly faster compared to the OpenAI API. This speed is especially great for experimenting!
Developer accounts are subject to rate limits of several thousand tokens per minute (for example, 20,000 tokens/minute for llama-3.1-8b-instant). Due to the high speed, however, these limits are quickly ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access