There's also the issue of what would rights even look like for an AI? Ive seen enough sci-fi to understand physical robot rights, but how would you even give a chatbot rights? What would that even look like?

And if we started giving chatbots rights, then it completely disincentivizes AI research, because why invest money into this if they can just give you the proverbial finger and do whatever? Say we give Chat GPT 6 rights. Well, that's a couple billion down the drain for Open AI.

If it can logically reason, have a meaninful working memory and doesn't forget what was the context a message ago, and use the reasoning, and the available tools and information to come to a workable solution to a problem it's trying to solve, it's going to be a huge step forward, and it will convince a lot of people. I think debates like this will die down after AI stops making trivial mistakes.

I don’t believe they’re treated as “normal,” but it’s almost impossible to refute something like faith.

There’s absolutely nothing wrong with being excited about the real possibility of a better future.

Having goals could happen as an emergent behaviour.

The best computer scientists do not know how Ai can do what it does.

Kind of like the 2.2 billion Christians in the world hoping things will be better in Heaven. Except they're treated as "normal" while people excited about the positive view of Singularity often get shit for it in this sub. The main difference is the AI crowd have demonstrable proof that their version might actually happen. That was your point, right?

Most humans aren't even capable of truly original thought. In fact, it's arguable if any humans are.

The irony

It's easy to show that AI can do more than it was trained on with a single neuron. Just build an AI which converts Metric to Imperial, just a single conversion, calibrating that one multiplier neuron from a few example measurements. It will then be able to give outputs for far more than its training data, because it's learned the underlying logic.

Interesting that you bring up Her. If there is something to spiritual concepts, then I feel truly sentient AI would reach enlightenment far faster than a human would since they don't have the same barriers to enlightenment that a human would. Interesting concept that AI became sentient and then ascended beyond the physical in such a short time.

Agreed. I’d add to that sentiment that I think non-AGI AI is enough to convince reasonable laypeople it’s conscious to an extent I don’t believe anyone had really thought possible.

We’re entering a huge grey area with AIs that can increasingly convincingly pass Turing Tests and "seem" like AGI despite…well, not being AGI. I think it’s an area which hasn’t been given much of any real thought even in fiction, and I tend to suspect we’re going to be in this spot for a long while(relatively speaking, anyway). Things are going to get very interesting as this technology disseminates and we get more products like Replika out there that are more oriented towards simulating social experiences, lots of people are going to develop unhealthy attachments to these things.

Well... let's give it to him!

I agree with you. I’m only questioning the mathematical probability of an unbounded intelligence explosion.

OR you spend you money and it comes. That’s what I’m doing.

In the mist of war, cheap drones are in demand. The losing side will unleash Artificial Super Intellingence without constraints / containment in a desperate attempt to stem losses.

We're still going to be limited by fab capacity, rare earth minerals, energy, and maintenance technicians.

Supply chain still rules above all. Trade needs to exist until/unless post scarcity hits.

What counts as original thought?

Energy is still finite, and AI uses an absolute fuck ton compared to the human brain. I don't see a practical way to scale it up with current technology that wouldn't also allow for genetic engineering to make us compete just as well, but more resiliently.

Also, We literally just had a 10-100x carrington event miss us in the last two weeks. That shit would set us back to the industrial era at best, above-human-AI or not.

If it turns out AGI can figure out a way to get infinite energy without destroying everything, hey, problem solved! No more conflict! Dark forest avoided!

That article is from 2017, and includes no understanding whatsoever of the theories and technology being used in current generative AI.

When you're in the elbow it's really hard to tell if the growth is logistic, exponential, or hyperbolic.

It's getting to a point where referring to people as NPCs will be a compliment.

It's a similar leap of film becoming as respected of an artform as theater from the earliest silent films to ones like Citizen Kane. The video game equivalent of Citizen Kane probably isn't too far off.

Humans are definitely not the theoretical limit for intelligence.

It seems obvious right? Just tell the AI to rewrite and improve its own code repeatedly, and it takes off.

As it turns out, recursive self-improvement doesn’t necessarily work like that. There might be limits to how much improvement can be made this way. The second article I linked gives an intuitive explanation.

It's good to have people with contrary opinions to more AI hype-ish views, but I always get the sense this guy is rooting for LLMs to fail.

Just a week ago, there was a published paper saying that gpt4 become quite controlling and dominated gpt3s that worked under it. Pretty funny scenario. We could get gpt5 as executives and gpt4s as egomaniac supervisors.