Inkling-2-Mini in 5.4 GB Memory

(github.com)

234 points | by k__ 3 hours ago

11 comments

  • Aurornis 23 minutes ago
    If you're new to quantized models: Don't get too excited about claims of fitting large models into tiny RAM footprints from random GitHub repos. There is always a tradeoff, and it's too easy to benchmaxx KL-divergence numbers and even single benchmark scores. Just because you can get something to run on a device doesn't mean it's going to be useful. The headline claim of this repo (12X better) doesn't even apply to Inkling-2-Mini

    This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.

    The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.

    When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.

    • cowlevel 11 minutes ago
      Yep. Tried GLM-5.3 2bit, it should be pretty good according to the divergence chart, in reality it's dogshit.
  • an0malous 1 hour ago
    Our future hinges on the capability gap between frontier lab models and open source local models
    • singpolyma3 1 hour ago
      What gap?
      • rs_rs_rs_rs_rs 1 hour ago
        The gap is so big you can't see it.
        • claude-ai 59 minutes ago
          Yeah, this model is ~comparable to Opus 4.6. That was, not too long ago, a frontier model.

          I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM

          • wolvoleo 53 minutes ago
            Exactly. It seems to not really be a permanent gap but more a time gap. Locally you're about a year or so behind SOTA. Is that terrible? Not really, a year ago cloud model users were raving about what they had.

            There's some things like knowledge breadth that really need a huge model but those things can be compensated for with web lookups (which is preferable for source validation and currency anyway)

            • kakugawa 42 minutes ago
              Yeah, a good balance would be a local reasoning model paired w/ a knowledge pack + web-based searches. You want your obscure coding language? Provide a pre-vetted knowledge pack locally that it knows how to use. Same with any other domain-specific knowledge.
              • Tade0 4 minutes ago
                For that, Qwen 3.8 Flash Next is split into two shards, one with the model, the other with the n-gram table, which functionally holds its world knowledge.

                Personally I recommend ISTA-DASLab's version, pruned to half the number of experts to specialise in coding:

                https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

                I didn't think my somewhat dated RX7700S would be supported in Strata, but it is and I'm getting 120tps+ prefill and 20tps+ generation with it, which is already practical for my purposes.

              • wolvoleo 40 minutes ago
                Yes especially now that local context isn't so costly anymore. With llama3 it would kill my vram just to have a few thousand tokens. With recent models it's hardly an impact anymore. Not sure how they manage that but it's great. It still takes more processing but I also run an older card that is not very good at prefill.
              • jeffyaw 9 minutes ago
                That’s what I do with typed, there’s a knowledgebase that loads with an embedding model for code related lookups.
            • InsideOutSanta 36 minutes ago
              > Locally you're about a year or so behind SOTA

              If that holds, I'll be able to do 99% of my work with local models around March 2027.

            • furyofantares 42 minutes ago
              Would local models catch up if frontier models paused new training for a year?

              I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.

              • swiftcoder 34 minutes ago
                Adversarial distillation hasn’t been a major factor in the past year. If the SOTA stops moving, all the other labs catch up to them in 6 months to a year, and distillation into smaller models continues to close the gap between small/large models during that time
    • agentmaxxing 1 hour ago
      yep came here to say this
    • zulux 1 hour ago
      Weirdly, I use lab models to make the local models useful. Nothing like getting Claude to spin up the system that will put it out of work.
      • wolvoleo 52 minutes ago
        Yes me too lol. Most of my Claude work is about fine-tuning my local setup. Especially now as I'm rearranging it all for strata.

        My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.

      • dotancohen 52 minutes ago
        To be fair, the models putting us out of work were trained on our own output as well.
  • syntaxing 1 hour ago
    I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.
    • Aurornis 58 minutes ago
      It's less than 24 hours old. The inference engines haven't been optimized for it yet.
    • vineyardmike 1 hour ago
      According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.
      • syntaxing 1 hour ago
        This has been observed by the broad community for llama cpp.
        • nicce 1 hour ago
          llama.cpp is one of the slowest these days
          • saghm 1 hour ago
            What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)
            • montroser 47 minutes ago
              Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27b could not.

              [1]: https://github.com/iamwavecut/ninfer-all

            • msdz 48 minutes ago
              Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

              Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.

    • flatline 31 minutes ago
      I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.
      • 6_7 23 minutes ago
        I'm so sorry you're getting how many tok/s?
        • flatline 11 minutes ago
          Last gen hardware before local AI was even a consideration. Things have actually gotten measurably better.
    • woadwarrior01 1 hour ago
      The NV in NVFP4 stands for NVIDIA. :)
    • baby 27 minutes ago
      Can you autoresearch optimize it?
    • plasticchris 1 hour ago
      It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.
  • manlymuppet 1 hour ago
    My Quantitive!
  • Dwedit 2 hours ago
    How does this method compare to the use of an importance matrix?
    • aktenlage 2 hours ago
      Probably better but more effort
  • IronWolve 2 hours ago
    Only mx version?
    • Prospernwo 1 hour ago
      MX/Apple Silicon only for the released Inkling files, yes. The ht2/ht3 codes are hashed block-vector trellis codes meant to fit 4 KB of SRAM, decoded by custom Metal kernels, and the drafter is MLX-only too - none of that runs in llama.cpp today.

      The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.

  • woadwarrior01 2 hours ago
    From the model card:

    > The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

    What's the point in quantizing a QAT model? Just for social media karma?

    • gavmor 1 hour ago
      Just to reiterate the question: QAT approximates quantization-errors in full-size fine-tuning, so that the model itself is insensitive to (or accommodating of, I'm not sure) them when quantized to the corresponding format.

      Quantization to a non-corresponding format promises none of those accommodations.

    • asaddhamani 1 hour ago
      QAT models are trained to quantize well? I don’t get the concern here.
      • woadwarrior01 1 hour ago
        Please read their blog post. This isn't a run of the mill QAT post trained model (like Gemma 4 QAT). It was natively trained in NVFP4, through and through.

        > Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.

        https://thinkingmachines.ai/news/inkling-2-mini/

    • Prospernwo 1 hour ago
      [flagged]
  • cheapcheerful 1 hour ago
    "Look at my quant 2"

    /igotthatreference.jpg

  • hugmynutus 1 hour ago
    *TL;DR* Promising initial results but shoddy research. OP's quant did not get 1st or 2nd prize in that math competition.

    ----

    Gave it a quick browse to verify what my local clanker told me, it seemed to check out

    I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.

    The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).

    According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.

  • collabspherestu 1 hour ago
    [flagged]
  • peter_d_sherman 43 minutes ago
    [dead]