This week, IBM’s new super computer, “Watson”, dismantled two former champions in a two-round, three-episode mini tournament on the popular quiz show, Jeopardy!.
The computer, which IBM trumpets as a major advancement in machines’ efforts to understand human language, boasts a nearly 3,000-computer-processor “brain,” which can perform various tasks simultaneously, and a program designed to decipher and understand the often very nuanced structure and abstract meanings within spoken and written communication (English in this case).
On the show, Watson received the clues through digital texts and then buzzed in against the two other contestants like any other player would. . . . Well, actually it buzzed in with a mechanical “thumb” instead of flesh and blood finger, but why split hairs?
After two days, the machine had accumulated 50,000 more points than the humans. The match was a rout.
Surrounding (and leading up to) this spectacle, there was a lot of hype by IBM about Watson and the potential for its program to be a major advance in artificial intelligence. Although it should be noted that IBM doesn’t yet give any specifics about what the advance(s) will be. And outsiders are divided into various camps over the specific potential applications of the program. But Watson has garnered a lot of TV air time and Internet space lately.
So what do you think?
Has Watson been over-hyped? Is this wishful thinking by IBM and AI enthusiasts? What application can you imagine for such a program? Are the chess-playing machines and Jeopardy!-playing machines the first wave of artificial intelligence, or are they still too rudimentary to be considered as such? Was the Jeopardy! tournament a fair match, with the best contestant winning? Other thoughts? Opinions?
The Afternoon Inquisition (or AI) is a question posed to you, the Skepchick community. Look for it to appear Tuesdays, Thursdays, Saturdays, and Sundays at 3pm ET.
AI is an interesting topic, and I suppose whether or not Watson qualifies somewhat depends on how it works. If it’s simply making keyword searches in databases, going for the highest fidelity to the “answer” given, I don’t see it as being intelligent so much as sophisticated. Because of it’s extremely limited usage (i.e. it’s not really terribly adaptable to anything other than playing Jeopardy), it’s more on par with an ant or a bower bird… capable of interesting and even somewhat incredible feats, but unable to really go beyond what it is programmed for.
The true measure of intelligence, in my mind, comes from self-programming ability. If something can pick up a skill through imitation and experimentation, it has a degree of intelligence. If it’s simply extremely capable of what it is programmed to do, it is simply extremely capable.
On a side note, I had to go back and correct references to Watson from “him” to “it”. I also find it really interesting the degree to which we anthropomorphize machines (such as the fact that all my cars have had names… from The Behemoth until the current Barry Allen).
I agree with Mark about what Watson really is and its applications in the real world. Its a great source/interpreter of information, not much better than a search engine now.
I enjoyed the shows and the previously aired PBS shows. My 7 yo loved it. We did refer to Watson as “he” during the shows.
That said, I don’t really think this was over-hyped. It’s been a long time since IBM got some kudos. (full disclosure: I was an IBM baby. My father was an EE with them back when the company was a big deal.) Look at all the hype Google, Apple and even Facebook get, why not allow a little for Big Blue. I can imagine being on the team that created Watson and what fun that must have been. Let them have their day in the sun.
As a professional programmer and an AI enthusiast I think this is a huge advance for computer science and really cannot be over-hyped. This is verging on the Star Trek computer and I am all atingle. The problem up until now has not been getting computers to come up with answers, but rather to understand questions with all the subtlety that language affords. Watson attempts to process questions, not as a single database query, but as many, possibly related, simultaneous queries, looking for connections between them, and trying to answer the question that makes the most sense. This is not too dissimilar to what the human brain does when processing language.
The uses of the technology are unbounded. Medical diagnosis comes to find first. Searching through scientific literature is another. How many centuries have been wasted collectively answering “Has this experiment been done before?” This technology can be applied to any question where the answer comes from stored information, but the question is more complicated than a google search. I for will be even more impressed when a Watson-like computer can understand and invent puns.
As @Mark Hall said there are more steps to be taken, for sure. Watson would be more useful if it could learn to not repeat mistakes.
@davew: I saw a mention, on an article about Watson (about the point where Ken Jennings answered “the 1920s” only to be followed by Watson doing the same”, that they honestly had not considered NEEDING to have it listen to wrong answers, and come up with the next best. I do find your mention of the Star Trek computers interesting, though… that you can say “Computer, BLAH” and it will correctly interpret your blah as language… that’s pretty awesome.
I also maintain that we should call Watson “she”, since it was obviously named after Rebecca. #brownnose
@Mark Hall:
But the voice they gave it sounded more HAL 9000 to me.
@Sam Ogden: Didn’t watch Jeopardy. My co-workers are more the “Family Feud” types, and I was going through a friend’s master’s thesis for lunch.
@Mark Hall:
Before the Jeopardy rounds started and in between them they had quite a bit of expose on Watson; interviews with IBM’s people, etc. I’m trying to recall if the developers referred to it as “He” or “It”. Let’s go to the video tape.
I think the most interesting part of this is what Watson did wrong. It failed to take into account a qualifier in the subject for one category “American Cities”. Another mistake it made was to repeat a wrong answer already given by another contestant. Sure, humans could make these same mistakes but the reason Watson did was that the programmers didn’t account for these situations. Unlike a human, Watson will continue to make those same mistakes over and over until someone figures out how to program it not to.
Something I don’t think anyone tried is re-asking it a question it already got wrong. A human would take another guess or give up. I’m willing to bet Watson would repeat the same wrong answer over and over.
Watson’s an amazing probabilistic natural language search engine but I don’t think that would count as intelligent. There are other projects going on that are working at the problem from the opposite end, developing machines that can learn from sensory input. At some point, these two types of projects will begin to merge and then I think we’ll start seeing something interesting.
@Steve D:
I think it was actually “US Cities”. One could almost see how it could get its answer of “Toronto” if the category was “American Cities”. So, yeah, it ignored the qualifier completely.
I wonder if I’ll ever see a super-intelligent computer given a feminine name in my lifetime. Unfortunately, I doubt it. As for this watson thing, I guess I should hit up youtube. My coworkers who watch jeopardy have been talking about it quite a bit.
So, he can understand language, but he had the clues texted to him? If that’s the case, then I don’t see this as being that important. He has to listen to the clues, then interpret the clues, then come up with a response, in order for me to consider it important.
I find this to be incredibly exciting. As @davew already said, Watson is able to understand language on a level not previously seen in an AI. What historically has always been the biggest challenge with artificial intelligence is being able to replicate the human brain’s ability to quickly and accurately surmise meaning in the spoken word, and to answer that meaning with an appropriate response. Watson has been able to not only understand what is said, but to also understand complicated nuance and implications… which is just extraordinary. Now, of course, the way he does this is through a vast number of relational databases, but honestly, that really is all the human brain is, isn’t it? From what I understand, though, IBM didn’t populate Watson’s “brain” entirely… they taught him how to learn – how to determine an answer based entirely on deductive reasoning… it’s really pretty exciting. (Sorry if I sound like a squirrelly teenage girl.)
On the flip side of my excitement, though, is a little bit of trepidation. After all, this is one step closer to the slow obliteration of an active middle class. Not that we have an active middle class winning rounds of jeopardy, but more, the advances Watson represents in AI communication and reasoning take us one step closer to machines replacing us in an already unstable economic landscape. (Watson took our jerrrrrb!)
This concern doesn’t mean I believe we should stop these exciting advances in science, just that it’s something to think about. Where would our college kids work if all our fast food joints are staffed by robots? I’m not sure I could enjoy my cheeseburger without the surly teenager serving it to me… ;-)
I’ll add to what Steve said; watching Watson from an academic perspective, and seeing what thought processes it was going through and the possible answers it was coming up with was just fascinating. Watson scored best when there was a lot of specific information in the question – I am sure this helped in finding enough material to answer the questions.
Many questions, Watson just didn’t have enough information to classify answers as having a high enough probability of being right to buzz in, yet the contestants were getting many of those same questions right. You could always see at the bottom of the screen Watson’s top three possibilities and the relative probabilities it was assigning them.
And I would just clarify the point about the US Cities question: firstly, it was the Final Jeopardy question, so each contestant HAD to provide an answer, unlike other questions where you could buzz in; and Watson had assigned the highest probability to Toronto at 14% – which was way below the cutoff at which it would have buzzed in on another question. So I don’t think this one stands out all that much – there were many other examples of Watson not having a high-enough probability answer to buzz in throughout the series, but this one seems to have been overly picked-up because it was the final question.
Watson had a fast finger (and that’s oooold technology), and that’s the only reason he won. That and given all word play, anagram and suggestive idiom types of categories were avoided because Watson would have been owned in those areas. IMO it was a brute force processing show with some job specific software aimed at getting the best possible answer. Kinda like a self contained Google search computer with Jeopardy in mind. No one thinks Google search engines are intelligent do they?
@James Fox: Fair point. I wonder if the scores would have been closer had there been a fair mix of word play, anagram, and suggestive idiom categories, as well as a bit of code in Watson’s motor skills program to replicate the average reaction lag between thought and action in a human.
The software itself has some remarkable implications, but it’s not quite AI. When they successfully demonstrate a program with Watson’s ability to answer questions, that can determine when idiom, sarcasm, puns, etc. are used even if it can’t understand the meaning, that can identify when a human is happy, or upset, or angry and give the right verbal response (even though it’s rote)…
Basically, I think I’ll be amazed by AI when it can not only understand human speech and retrieve data in response to questions, but can perform socially at least as well as a high-functioning autistic adult of average intelligence.
@James Fox: Kinda like a self contained Google search computer with Jeopardy in mind. No one thinks Google search engines are intelligent do they?
I see it as a step towards passing the Turing test. How big a step I admit is a matter of debate.
I, for one, welcome our new cyborg overlords.
Alternate comment: sounds like they hooked Google up to a button. Turing test as I understand it is to determine if a human could tell the machine apart from a human…if that’s correct, then Watson is not even a small step cloer to that. It interpreted speech and spit out answers without “thinking” about it, just correlating facts from a database. It’s closer to a pile of encyclopedias than it is to a human.
The kinds of questions that would actually require thought or creative thinking are the ones that were specifically not asked. Rigged contest, really.
James Fox,
I’m not so sure it’s useless at word play. From a NY times article back in june, when they were still working on it, it could answer questions like: “Classic candy bar that’s a female Supreme Court justice” — “What is Baby Ruth Ginsburg?”. Maybe that isn’t the most devious word play ever, but it’s certainly not just a search engine. Obviously it’s not AI, and it wasn’t devised to pass the turing test, but the language comprehension and the ability to combine various threads of information in a meaningful way has me seriously impressed.
@unexplainedbacon: Some would say human brains are simply a pile of encyclopedias… sure, encyclopedias of compromised data, potentially volatile, and certainly not always normalized… but data all the same.
No, it’s not perfect… but it can pick up very simple nuance (as ethanol points out, and by the very fact that Jeopardy is known for its word play – albeit basic – and he still won).
Of course Watson still has a lot of growing to do… but he is still a big step towards true AI.
@James Fox
If you think a “fast finger” was the only reason Watson won, you are incredibly ignorant of the limitations of digital computers.
Speech recognition alone presents one hell of a problem. Computers have a lot of trouble even detecting the breaks between words – a classic example used in the field is “how to wreck a nice beach” (how to recognise speech).
Even for just the speech recognition aspect, this is a huge advancement. Like davew said – this simply cannot be over-hyped.
Top marks to IBM. Hopefully in a few years, Apple will copy it, make it look cool, and pretend they invented it. Then it can go mainstream ;)
It’s an interesting development, I’ll grant you that. Other than that, I’m really not qualified to give an opinion of its significance, as I’m not an IT professional. I really don’t understand how it works on a deep level. I gather that it’s something more than a glorified Google from the comments above?
@davew: I have a very basic problem with the Turing test. Namely, I can be face to face with a human being, and still not be sure if I’m dealing with an intelligence.
Speech recognition is really not a part of the Jeopardy! format. I’ve been. The studio is not as large as the cameras make it look, and the contestants are reading the clues right off the board. The only reason we were even listening to Alex is to help with the timing on the buzzers. As soon as Alex finishes reading a clue a light goes on (not visible on camera) indicating that the buzzers are armed. From there the challenge is reaction time, since ringing in too early deactivates your buzzer for 0.5 sec., which on this scale is a really long time.
I have not heard what system Watson used for this. Did he have a visual sensor for the light, or was he given an electronic signal for it?
@Ubi Dubium:
Cool. I always wondered about the process in studio.
The screen behind the contestant podium was just there to display Watson’s avatar. Watson’s hardware filled an entire room (not sure of the size though substantial) somewhere outside of the television studio. And they only briefly showed its ring-in mechanism, and they weren’t specific about where it was located or how Watson was signaled that it was okay to ring in.
@darrenc: @Sam Ogden: It was just like any other Jeopardy game. A light goes on when AT is done reading the answer and only then can you press the button. If you ring in early your button is reset with a quarter second penalty delay. A human can get good at anticipating this but a machine who has the answer will beat the human almost every time except when the human gets lucky. IBM built in a small delay when Watson had less confidence in a particular answer in an attempt to mimic a human reaction; but as we saw when Watson was sure enough his digital speed advantage was insurmountable. And don’t me wrong, I think IBM did something quite interesting and novel; I just don’t think it was ground breaking or amazing that Watson was able to win at Jeopardy.
Also from what I’ve read Watson did not receive the clues via voice recognition. (Now apparently confirmed by IBM) Watson read them digitally thus giving him time to complete his searching and analyzing long before the light came on allowing the button to be pushed.
@darrenc: “Time flies like an arrow, but fruit flies like a banana.”
While I think it’s interesting, and computers have come a long way, I’m still not terribly impressed. Of course, I’m harsh, and think that they totally softballed the questions by not putting in more difficult categories like: before and after, rhyme time, and other “less than straight trivia, plug and chug” sort of questions.
I also think they softballed it by not having the computer be able to read the clue itself, or have to listen to Alex. Why should it get special treatment?
The computer is an it, please don’t refer to it as he, it is not a he, nor is it human. When it can ask to be treated like a person, it will be. Don’t genderize things with no gender.
The Baby Ruth thing is a perfect example of simple corrolation vs. any kind of thought or creativity or evne interpretation. That’s not only an easy gimmick, I’d be very surprised if it’s anything that hasn’t been done long, long ago. It is, again, simple Google-smarts. Nothing to report.
Also, before, I refered to it interpreting speech. Turns out it didn’t even do that, it has to be fed the questions (well answers due to the game format) in its own special AI-talk. I bet it would be even less impressive if we knew the details of that input.
As far as the human brain being just a pile of encyclopedias, you could definitely say that, you’d just be dead wrong and vastly, unspeakably underestimating all the really goes on in the brain. Pretty much none of which Watson is capable of. Being able to collate random factoids is part of what the brain does, yes, in a rudimentary, tip-of-the-iceburg way.
The sinple harsh reality is, despite how appealing and fun the idea of AI is, we don’t even understand more than the tiniest sliver of what OUR brains do, let alone know how to reproduce anything even contemplating a similar level of intelligence or thought. We’ve learned more about the brain in the last 10 years than in the entirety of human history combined and we will need to go so, so much further before we can consider reproducing that.
Unless you have a uterus and some sperm, in which case it’s really pretty simple =)
@unexplainedbacon: Like UB and Ken Jennings, I welcome our robot overlords. As tailored as the process may have been for this contest, it seems like Asimov got it right some 60 years ago… We had better hard-wire into our creations some subservience to humanity. A properly designed computer or robot has a good chance of surpassing a human at a specific task. The next potential problem is whether (when) they will surpass humans in a wide set of tasks.
Then again, I just finished re-watching BSG, so maybe that had an effect.
@BlackCat:
I expect there may be a struggle to get this to happen. Lots of the Asimov robot stories were about this fight. (On the surface, anyway. Underneath, they were stories about the perpetual human struggle against bias and prejudice and jumping to easy conclusions.)
I’d like to present another angle on this: A machine has yet again become better than the best human at a task that machines couldn’t even concievably do a decade ago.
Who’s excited? Who’s scared? Who’s out of a job?