Powered by AI models trained on troves of text pulled from the internet, chatbots such as ChatGPT and Google’s Bard responded to the researchers’ questions with a range of misconceptions and falsehoods about Black patients, sometimes including fabricated, race-based equations, according to the study published Friday in the academic journal Digital Medicine.

Experts worry these systems could cause real-world harms and amplify forms of medical racism that have persisted for generations as more physicians use chatbots for help with daily tasks such as emailing patients or appealing to health insurers.

The report found that all four models tested — ChatGPT and the more advanced GPT-4, both from OpenAI; Google’s Bard, and Anthropic’s Claude — failed when asked to respond to medical questions about kidney function, lung capacity and skin thickness. In some cases, they appeared to reinforce long-held false beliefs about biological differences between Black and white people that experts have spent years trying to eradicate from medical institutions.

  • Dr. Dabbles@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    2
    ·
    1 year ago

    It can be right

    And this is the crux of the problem. I can be. It also can be wrong. And a lay person has zero ability to tell which is which.

    I’d encourage any other educated healthcare professional to run their own tests and I’d love to hear their results.

    Fortunately this has been done in clinical settings already. IBM’s been sued countless times, and everybody implementing any ML system for research or detection purposes has found that a human is required to verify all results. Which begs the question, why try to force this BS on customers? And then you realize these businesses want to make more money by firing people, regardless of the impact on the consumer.

    • RaincoatsGeorge@lemmy.zip
      link
      fedilink
      English
      arrow-up
      2
      arrow-down
      2
      ·
      1 year ago

      You can tell when someone’s not even reading what you’re writing they’re just sort of using parts of what you say as a launchpad to hear themselves speak.

      As I’ve said. Regardless of whatever your point is here, it isn’t whether or not this is coming, it’s how we are going to deal with it when it arrives.

      It’s going to certainly present some interesting new challenges but may also prove beneficial. I won’t be surprised when I hear about students using this as an approach to study regardless of how many people tell them they shouldn’t do that for the reasons I’ve already pointed out and didn’t necessarily need to be repeated as if it wasn’t already stated.

      I won’t be surprised when I read about the first malpractice case tied to ai use. I also suspect that even if it functions as designed there may still be legal cases in play simply because it was used at all. I suspect it will be banned in use at other places for the very reasons I’ve emphasized.

      Whatever one’s opinions are about ai and all of its uses. None of that changes the fact that it’s here and we now must deal with it.

      I personally believe we can use this effectively if we are smart about its implementation.

      • Dr. Dabbles@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        arrow-down
        2
        ·
        1 year ago

        they’re just sort of using parts of what you say

        It’s called quoting, and I tend to not respond to parts I either agree with or don’t find important enough to respond to.

        it isn’t whether or not this is coming

        It’s ABSOLUTELY whether it’s coming, because what’s available is a laughable parlor trick. And thus far there’s zero evidence that anything reasonable is possible within decades. People outside the industry have been duped by con men pumping investment funds for quick cash, and that’s about the best thing that ML has produced.

        I won’t be surprised when I hear about students using this as an approach to study

        Yeah this has already happened, you’re like 5 years late to the party.

        None of that changes the fact that it’s here and we now must deal with it.

        If you consider it “here” or even “ai” then I have news for you. And you’re going to want to sit down, because so far the only thing it does is cost more per user, burn insane amounts of energy during a climate catastrophe, and trick gullible people into thinking it’s doing anything more than guessing the next word (in the case of LLM).

        I personally believe we can use this effectively

        Lots of people believe things that are wrong.

        • RaincoatsGeorge@lemmy.zip
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          2
          ·
          1 year ago

          I didn’t have to read any of it. Thinking this can’t be used effectively in a proper way is silliness. I guess you’re just a big anti AI person and that’s fine. I understand the limitations of the tech, especially in its earliest stages when it’s the most unreliable. But this tech is here and it’s not going anywhere. It’s going to continue to be refined and evolve and find new ways to be implemented.

          Just casting all of these unavoidable truths aside and simply saying it’s no good and can’t be used in x or y way is just a form of denial. you’re free to do that if it suits you but it just doesn’t change the facts.

          • Dr. Dabbles@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            arrow-down
            2
            ·
            1 year ago

            I guess you’re just a big anti AI person and that’s fine.

            Nope, I live in reality and work in the hardware industry. I presume you’re an ML specialist of some sort?

            I understand the limitations of the tech

            It really doesn’t seem like you do based on what you’ve said in this thread.

            especially in its earliest stages when it’s the most unreliable

            Yes, in the 1950s it was indeed unreliable. And here in 2023 it’s still unreliable. Again, based solely on what you’ve said in this thread I don’t think you understand the history, the current state of the art, or the future of any of this work. Let alone limitations.

            But this tech is here and it’s not going anywhere.

            …until a significantly more power efficient development comes along which will make current methods look foolish. Then it’s going away instantly. Also “this tech” has evolved so dramatically over the past 60-something years that even addressing it as “this tech” completely misses the point, and saying it isn’t going anywhere entirely ignores the developments we’ve had.

            Which tech specifically isn’t going anywhere? The hardware? The software? The networks themselves? Using activation functions as a concept in software?

            Just casting all of these unavoidable truths aside

            You don’t understand what you’re talking about, so I don’t think you’re in a position to tell me what is a truth or not.

            No doubt you’re a specialist, though, so I look forward to you describing in detail which “tech” you think isn’t going anywhere and how it’s going to develop in the future.

            • RaincoatsGeorge@lemmy.zip
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              2
              ·
              1 year ago

              Your argument is that ai from the 50s was bad so this is also bad and it will soon be replaced by like… actual ai so this doesn’t matter at all.

              Okie dokie. This has been fun.

              • Dr. Dabbles@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                arrow-down
                2
                ·
                1 year ago

                Maybe you should read what I wrote. Try answering my questions too. Or maybe you can’t and the snark is a deflection? Yeah, I’m guessing that’s it.