Persona of the day: BWOL The Kernel Hacker (UR, Legendary).
Builds with open models and only trusts the stack they set up themselves.
"You don't use AI. You compile it."
No card has landed on it yet. Yours could be the first.
@my9ai.com
@my9ai.comPersona of the day: BWOL The Kernel Hacker (UR, Legendary).
Builds with open models and only trusts the stack they set up themselves.
"You don't use AI. You compile it."
No card has landed on it yet. Yours could be the first.
@abchaudary.meA dense model reads every weight once per generated token, so tokens per second tops out near bandwidth divided by model size. Intel has not published the D270's bandwidth yet.
If you run models locally: which limit do you hit first, fitting or speed?
@abchaudary.me
@amz34.bsky.socialLocal LLMs now run on everyday hardware. Niko1221/Strata: MIT-licensed C++ inference engine, one-click Windows/Linux install, OpenAI-compatible API on localhost.
github.com/Niko1221/Strata
https://awesome-agent-infra.pages.dev/?utm_source=bluesky
#AI #OpenSource #LocalLLM
*evolution noises* moving from built-in #ollama UI to THE #OpenWebUI !!ミ＼(≧ﾛ≦＼)
honestly, decided to adopt it out of annoyance at lack of dark theme in ollama's #UI 😅
but now looking forward to explore Modelfiles, project folders, search in chats and RAG 🤩
@miniai.bsky.socialIntroducing MiniAI Quata2 4B.
This is the first of the 3 Quata2 series models that will come out.
The model will finish uploading in a couple of hours, if it does not finish uploading today i will check to upload it tomorrow.
@jrone89.bsky.socialYou don't need a rack of servers to run an AI assistant https://pragmaticsysadmin.help/sysadmin/2026-10-05-local-llms-as-system-tooling-running-many-sub-3b/ #ai #localllm #sysadmin
@logeshwaranorg.bsky.social
@alexraustin.com@logeshwaranorg.bsky.socialA #speech-to-text model smaller than one phone photo.
Whistle is 16.9 MB, runs on any CPU with no internet, and beats Whisper base (145 MB) on LibriSpeech. #AI #LocalLLM
The catches: 7 languages, 30 s per pass. Install guide:
www.logeshwaran.org/2026/10/cact...
@alexiskirke.bsky.social27.4B agent decision model now runs local from 17.67 GB instead of being trapped on giant servers.
ggml-org/OpenJev-GGUF
https://huggingface.co/ggml-org/OpenJev-GGUF
@jrone89.bsky.socialYou need an AI assistant that runs on your own machine https://pragmaticsysadmin.help/senior-tech/2026-10-05-local-llms-as-system-tooling-running-sub-3b-models/ #ai #localllm #privacy
@logeshwaranorg.bsky.social#Google just released #EmbeddingGemma 2: one open 740M model for text, code, images, video and audio in one search space. 378 MB text-only in Ollama, Apache 2.0, runs on a phone.
www.logeshwaran.org/2026/10/embe... #AI #Opensource #LocalLLM
@tony10000.bsky.social@tony10000.bsky.social
@botmonster.comCuts wasted retrieval calls ~35%; 30-40% of chatbot queries need none.
HyDE embeds a fake answer paragraph, lifting recall 10-15%.
https://botmonster.com/ai/agentic-rag-llm-decide-when-what-to-retrieve/?utm_source=bluesky&utm_medium=social #LocalLLM
@w512.bsky.socialCan a 125B model run on a 16GB GPU and still do real work?
I ran Qwen3.8-Flash-Next with Strata on an RTX 4060 Ti: ~60–70 tok/s, and it solved 2 of my 3 coding tasks. Catch: you need 64GB of RAM.
@shawonshovon.bsky.socialSidekick is the kind of project that gets better the longer you run it.
Sidekick is a open-source local llm. Chat with a local LLM on macOS using your files — no extra setup.
→ Full source available on the repo link.
→ Released under open-s...
@recepcinet.bsky.social@logeshwaranorg.bsky.social