Way to whip that LLaMA's ass
A handy open source tool for packaging up LLMs into single universal chatbot executables that are easy to distribute and run has apparently had a 30 to 500 percent CPU performance boost on x86 and Arm systems.
You can build and run llama.cpp natively, give it a model to load, and then interact with that LLM in various ways. Where it gets tricky is the model files involved are usually quite large in size, and it can be a bit confusing to know which variant is best to use. For instance, on an Intel Skylake Core i9-9900, prompt processing jumped 50 percent versus llama.cpp, whereas evaluation stayed the same.
While the performance gains that come with MKL are great, the fact that it's closed source is less than ideal for this open source effort, according to Tunney. She noted that"integrating foreign BLAS libraries into llama.cpp isn't that practical, due to the way its threading model works." And since MKL is closed source, it's not possible to just look at it and see how it can be improved.
United Kingdom Latest News, United Kingdom Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Databricks claims its open source foundational LLM outsmarts GPT-3.5In the AI gold rush, analytics outfit wants to provide the shovels
Read more »
Can generative AI truly transform healthcare into a more personalized experience?The current literature on large language model (LLM)-based evaluation metrics for healthcare chatbots.
Read more »
You got legal trouble? Better call SauLM-7BCooked in a math lab, here's an open source LLM that knows the law
Read more »
Intel is still top dog: ships 3x more CPUs than AMD and Apple combinedAndy built his first gaming PC at the tender age of 12, when IDE cables were a thing and high resolution wasn't. After spending over 15 years in the production industry overseeing a variety of live and recorded projects, he started writing his own PC hardware blog for a year in the hope that people might send him things. Sometimes they did.
Read more »
First Armv9 automotive CPUs aim to power AI-enabled vehiclesVehicle electronics and software becoming ever more complex
Read more »
Beijing issues list of approved CPUs – with no Intel or AMD2024 may be the year of Linux On The Arm-or-RISC-desktop as China moves away from Western tech
Read more »