(this account)

Stump. I was a tree, but now a dump.

llama.cpp is at the core of almost all offline, open weights models. The server it creates is Open AI API compatible. Oobabooga Textgen WebUI is more user GUI oriented but based on llama.cpp. Oobabooga has the setup for loading models with a split workload between the CPU and GPU which makes larger gguf quantized models possible to run. Llama.cpp, has this feature, Oobabooga implements it. The model loading settings and softmax sampling settings take some trial and error to dial in well. It helps if you have a way of monitoring GPU memory usage in real time. Like I use a script that appends my terminal window title bar with GPU memory usage until inference time.

Ollama is another common project people use for offline open weights models, and it also runs on top of llama.cpp. It is a lot easier to get started in some instances and several projects use Ollama as a baseline for “Hello World!” type stuff. It has pretty good model loading and softmax settings without any fuss, but it does this at the expense of only running on GPU or CPU but never both in a split workload. This may seem great at first, but if you never experience running much larger quantized models in the 30B-140B range, you are unlikely to have success or a positive experience overall. The much smaller models in the 4B-14B range are all that are likely to run fast enough on your hardware AND completely load in your GPU memory if you only have 8GB-24GB. Most of the newer models are actually Mixture of Experts architectures. This means it is like loading ~7 models initially, but then only inferencing two of them at any one time. All you need is the system memory or the Deepspeed package (uses disk drive for excess space required) to load these larger models. Larger quantized models are much much smarter and more capable. You also need llama.cpp if you want to use function calling for agentic behaviors. Look into the agentic API and pull history in this area of llama.cpp before selecting what models to test in depth.

Huggingface is the goto website for sharing and sourcing models. That is heavily integrated with GitHub, so it is probably as toxic long term, but I do not know of a real FOSS alternative for that one. Hosting models is massive I/O for a server.

tendentious milieu, or mircomilieu

No country on Earth is like this. No polity is unified without opposition. Additionally, making this statement here, when it is obviously false, in a place where at least half the people are from the US, and likely all of those are left leaning and just as much victims if not more so, seems like rather tepid malevolence.

It is not a laser beam.

No. The primary way of blocking radio is by raising the noise floor across the band. The type of radio is irrelevant. The protocol is irrelevant. It is all only the electromagnetic spectrum from infrared light, to visible spectrum light, to radio light, through to xray or gamma ray light. How we divide that up into protocols, bands, and names is totally irrelevant. When transmitting radio light, we are all restricted in how much power we are allowed to send. All receiver circuits are listening for meaningful information above the noise floor. Bands are allocated to try to create spaces for certain types of communications. This controls the noise floor. Then electrical engineers design the hardware you buy to operate within this specification. If that noise floor is raised, the physical hardware is unable to retrieve information and effectively makes it useless. If you are a radio wizard and build your own transmitter that has more power, you just created a giant beacon that anyone will track easily to your location. Transmitting always reveals your exact location. In military operations, you constantly hear about some entity going radio silent. This is why. If you are a soldier, you may not carry a cell phone at all when on the job because it is constantly revealing your location. The only way to avoid this is with actually hard wire connections. You are able to use lasers for line of sight communications, but in practice, you will be limited by the optical lens focusing complexity and atmospheric distortion even from the ground with point to point regional communication. If anyone crosses the beam it will still be detected and is likely to leak some light depending on conditions and design.

Ultimately, your only real option is the sneaker net which is damn near useless in US suburbia hell. Don’t forget that the freeway system was not created for the citizenry. It is only about military mobility. That is why the Germans made the autobahn and why the USA and others had to copy the idea. Your only defense is in the democratic political space.

I am talking about something where there is no research done. No doctors exist in this space.

It doesn’t matter anyways. I found how the model’s last layer of thinking defense gets around the issue. I can turn off most of alignment, but cannot actually fully control it totally unchecked.

You assume much, and are being an ass in my opinion. Believe it or not, science is not always well funded. If you happen to be curious and have the time, it is possible to explore scientifically or even casually within areas that are not well researched. It is possible to have logic skills even without credentials.

We are not in some final state of technology. Anyone saying such nonsense lacks fundamental logic skills.

I do not care about me. I do not have dogma. I’m not interested in recognition. I am willingly to explore in unique ways both artistically as a professional artist, and out of logical curiosity. I have the tools needed to check my results against a control using unrelated sources. The most recent paper on the subject is something I can recreate but explain far better than that paper.

I could not care less what you ultimately think of me, or anything I say. What I care about is that you’re a decent digital neighbor. To be physically disabled in near total social isolation, and have a place like this as my main interaction with other humans, it is a mean prejudice to have some random digital neighbor make such unsolicited malevolent statements assuming my personal motivations without a shred of evidence or decency to engage in questioning. You know absolutely nothing about me, yet you presume a great deal, putting words to my emotions as if you own me.

Sometimes the whole world does seem crazy. So I’m not liking my odds. Thanks for the rational advice.

What if you’ve got no credentials, but the flaw is so serious that it will not matter if known.

This is a true hypothetical curiosity. I do not know anything of value. A bunch of people here like to call me crazy, and I’ve rambled on and on many times in ways that likely confirm their notions. A person like this is not likely to fair very well when operating well outside their social caste unless they already have hand holds on the rungs of the ladder above. Still, there are some rather surprising areas of technology without adequate fundamental research. Perhaps it is hypothetically better to have John Conner in the world of Cyberdyne. If someone had killed Apache early, the Internet would not be the same heaven of democracy, though that is not a very good intuitive scope of analogy. Just something to ponder if one were to be in such a situation.

They dropped out of computer science last week. Ask Anybody, they were there.

Billionaires. Governments are small pawns.

Thought he bubba babyin Bill?

All technology would instantly halt.

The actual bond wires between the silicon pad and chip packaging for every chip with some kind of leads (feet) is actually an extremely thin thread of pure gold. It has to do with the super tiny size of the actual pads that are being bonded on the die, the robotics, the welding, and the physical properties of the wire connection.

K&R?

Nah, I think people that have critical thinking skills are not afraid of being wrong, aka their own curiosity and growth. The rest of the world wants simple dichotomous logic and is incapable of constructive engagement. They are simply too narcissistic to process abstract thinking and are looking for any excuse to rot in their dystopian existence.

Graduated pacman emerges… and we all know emerge is Gentoo. This one doesn’t compile.