bcovertigo

Black hat and Defcon just ended and I’ll share my impression from LLM related talks given there. Microsoft VPs charged additional money to CISOs attending the summit talking about how AI will disrupt and be the future and blah blah magical thinking.

Meanwhile Microsoft engineers and others said things like “this is logarithmic regression for people who are bad at math, and is best for cases where 75% accuracy is good enough. Try to break use cases into as many steps as possible and keep the LLM away from any automation that could have any consequences. These systems have no separation between the control plane and user input, which is re-exposing us to problems that were solved 15 years ago.”

I think there are some neat possibilities that are lost in marketing hype as venture capitalist anger grows that they might have been scammed by yet another hammer in search of nails.

“Nut meat” is a common phrase so I would guess the peanut product is closest, but please stop this line of thought for your own safety.

Confusingly, there’s actually two similar staves that get mixed up. The helix patterned one with two winged snakes I think you have in mind is called the Caduceus, but the the single wingless version I meant is the staff of Aesculapius (multiple spellings out there).

Go check out the alledged link between the snake wrapped staff that’s used to represent medicine and the treatment for guinea worms. Googling puts that theory with the Ebers papyrus from 1500 BC if it’s true!

Art by Jesper Ejsing in case you want to see the wider frame version!

It’s valid to point out that we have difficulty defining knowledge, but the output from these machines are inconsistent at a conceptual level, and you can easily get them to contradict themselves in the spirit of being helpful.

If someone told you that a wheel can be made entirely of gas do you have confidence that they have a firm grasp of a wheel’s purpose? Tool use is a pretty widely agreed upon marker of intelligence and so not grasping the purpose of a thing that they can describe at great length and exhaustive detail, while also making boldly incorrect claims on occassion should raise an eyebrow.

Low effort speculation:

That’s a vodaphone portugal IP, but this is likely traffic routing though their customer cellular network and not their corporate. It’s possible that someone in PT has a similar username for this service and is fat fingering it. It’s also possible that you’re seeing a tiny sliver of a larger attack.

Spur.us tracks that IP as an egress point for openproxy and windscribe ResIP networks so it’s worth considering that the origin of the authentications you’re seeing may not be Portuguese cellphone but someone hiding behind those services.

Here’s a paper describing the difficulties such a service creates for folks trying to secure accounts with traditional IP reputation based rules. “Resident Evil: Understanding Residential IP Proxy as a Dark Service” https://ieeexplore.ieee.org/document/8835239

Shooting in the dark for how a bad actor would monetize account takeover for this service if this is in fact an attack… They could try to sell your invitation to that private tracker. They could also look to scoop up a bunch of folks to try and blackmail based on what victims are download/seeding. Other more creative options I’m not thinking of might be on the table.