Tootfinder

So to follow up on this, I've caught it in action. Models, when quantized a bit, just do a bit more poorly with short contexts. Even going from f32 (as trained) to bf16 (as usually run) to q8 tends to do okay for "normal" context windows. And q4 you start feeling like "this model is a little stupid and gets stuck sometimes” (it is! It's just that it's still mostly careening about in the space of "plausible" most of the time. Not good guesswork, but still in the zone). With long contexts, the probability of parameters collapsing to zero are higher, so the more context the more likelihood you are to see brokenness.
And then at Q2 (2 bits per parameter) or Q1, the model falls apart completely. Parameters collapse to zero easily. You start seeing "all work and no play makes jack a dull boy” sorts of behavior, with intense and unscrutinized repetition, followed by a hard stop when it just stops working.
And quantization is a parameter that a model vendor can turn relatively easily. (they have to regenerate the model from the base with more quantization, but it's a data transformation on the order of running a terabyte through a straightforward and fast process, not like training).
If you have 1000 customers and enough equipment to handle the requests of 700, going from bf16 to q8 is a no-brainer. Suddenly you can handle the load and have a little spare capacity. They get worse results, probably pay the same per token (or they're on a subscription that hides the cost anyway so you are even freer to make trade-offs. There's a reason that subscription products are kinda poorly described.)
It's also possible for them to vary this across a day: use models during quieter periods? Maybe you get an instance running a bf16 quantization. If you use it during a high use period? You get a Q4 model.
Or intelligent routing is possible. No idea if anyone is doing this, but if they monitor what you send a bit, and you generally shoot for an expensive model for simple requests? They could totally substitute a highly quantized version of the model to answer the question.
There are •so many tricks• that can be pulled here. Some of them very reasonable to make, some of them treading into outright misleading or fraudulent, and it's weirdly hard to draw the line between them.

Got sent this email from the elderly convenor of a political group I sometimes sit with. Thank goodness I was alert and suspicious enough - his account had been hacked and this was an email spam that could have cost a lot of money. There was no "Dear Lilly", but they had signed off with his name at the end.

Nations agree to release oil reserves as war in Iran hits global economy
The International Energy Agency on Wednesday announced that it would carry out its largest-ever release of oil reserves
— 400 million barrels
— in a bid to control spiking energy prices caused by the United States-Israel war against Iran.

The plan, aimed at stabilizing oil prices that have soared since the United States and Israel attacked Iran on Feb. 28, would deplete roughly one-third of glob…

Nations agree to release oil reserves as war in Iran hits global economy
The International Energy Agency announced that it would carry out its largest-ever release of oil reserves — 400 million barrels.

49ers now know what Maxx Crosby trade will cost as rumors continue to swirl https://www.sportingnews.com/us/nfl/san-francisco-49ers/news/49ers-now-know-what-maxx-crosby-trade-cost-rumors-continue…

This week’s post jumps from Super Bowl ticket prices to Oklahoma’s new steps toward transparency on asset forfeiture—and then into a long-running fight over casket sales and licensing. Sports, policy, and skepticism included. 🏈⚖️
https://www.bobmuellerwriter.com/foo…

Football Dreams, Government Overreach, and Small Businesses Pushing Back – Bob Mueller
Super Bowl prices, Oklahoma reforms, and a look at why asset forfeiture transparency and licensing laws matter to regular people.

"Adopting low-cost ‘healthy’ diets could cut food emissions by one-third"
#Diet #Emissions #Food

Adopting low-cost ‘healthy’ diets could cut food emissions by one-third - Carbon Brief
A new study, published in Nature Food, compares prices and emissions associated with 440 local food products in 171 countries.

Someone really tried to convince me---did they have shares?---to use LLMs:
"It's like having an extra junior programmer at your disposal!"
OMG, not another junior, please!
The only reason to waste your time on a junior programmer, is so that they might learn and grow faster, and in a couple of years the time they cost you, and you invested in them, may just start to pay off. It's always a gamble though. Who is worth "wasting" your precious…

Super Bowl LX expected to see highest ticket price since 2020 https://www.espn.com/nfl/story/_/id/47820271/tickets-super-bowl-lx-high-cost-seahawks-patriots-nfl

Eagles get updated Maxx Crosby trade value after wild Micah Parsons-like projection https://www.sportingnews.com/us/nfl/philadelphia-eagles/news/eagles-updated-maxx-crosby-trade-value-micah-parson…

“Of all the possible Middle East scenarios, 🔥the current state of play is one of the worst for the global economy,”
says the Commonwealth Bank of Australia’s head of global economics, Joseph Capurso.
He added: 💥“We expect the situation to escalate before it de-escalates.
“Iran’s leadership and military capabilities have been significantly degraded.
However, what is unknown is their intent and capability to block the