Streaming Model Weights for Enhanced Accessibility
A new system called Janas allows large language models (LLMs) to run on hardware with less RAM than the models themselves require. This is achieved by efficiently streaming model components from fast SSD storage. Janas demonstrated superior performance compared to existing solutions like llama.cpp on identical hardware.
Breaking news
Town AI Secures $108 Million Series B, Valued at $1 Billion After $55 M Pre‑Series
Apple Prepares for Major Product Launch Event on October 28
Apple Releases Release Candidates for watchOS, tvOS and visionOS 27.2
Apple Limits iPhone Duo and iPhone 18 Pro Max to Reservation-Only Sales in Japan and South KoreaThe breakthrough was showcased on a laptop with 32 GB of RAM. It successfully ran the Qwen3-Next-80B model, which is 48 GB in size. This was accomplished by intelligently loading parts of the model from the SSD as needed. This technique overcomes a significant barrier for deploying powerful AI models on consumer-grade devices.
Janas achieved impressive speeds during testing. It processed text at 37 tokens per second with the large Qwen3-Next-80B model. This was on a laptop equipped with an Intel Core Ultra 9 processor, integrated Arc graphics, and 32 GB of RAM. The system uses an NVMe SSD for storage.
Can LLMs Now Fit on Any Device?
In direct comparisons, Janas consistently outperformed llama.cpp. It handled prompt reading at 62 tokens per second, significantly faster than llama.cpp's 16 tokens per second. Even when running without caching intermediate computations (drafts), Janas maintained a speed of 24 tokens per second. This suggests a highly optimized data streaming pipeline.
The core innovation lies in Janas's method of managing model data. Instead of loading the entire model into RAM, it streams necessary sections from the solid-state drive. This process is optimized to minimize latency. The system intelligently predicts which parts of the model will be needed next.
This approach makes it feasible to run much larger and more capable LLMs on devices with limited memory. Previously, running such models required expensive hardware with vast amounts of RAM. Janas democratizes access to advanced AI capabilities. Users can now leverage powerful models on their personal laptops and desktops.
Frequently Asked Questions
What is Janas? Janas is a new system designed to run large language models. It enables models larger than the device's available RAM to operate efficiently.
How does Janas work? Janas streams model components from fast SSD storage directly into memory as needed. This avoids loading the entire model at once, reducing RAM requirements.
Is Janas faster than other solutions? Yes, Janas has shown faster performance than llama.cpp in direct comparisons on the same hardware. It achieves higher token processing speeds.
