Man vs App on Jeopardy tonight.

[Moonliner]

Tonight is the debut of "Watson" the Jeopardy playing app. (http://www.engadget.com/2011/02/14/humans-had-a-good-run-watson-to-debut-on-jeopardy-tonight/)

Today, tomorrow, and the 16th are the fateful days: IBM's Watson supercomputer will go head to head with Brad Rutter and Ken Jennings in this silly, human-devised game we call Jeopardy. It promises to be some kickass TV, at the very least, and a historic event if Watson can prevail over his fleshy competition.

Here is video of the practice round. (http://www.engadget.com/2011/01/13/ibms-watson-supercomputer-destroys-all-humans-in-jeopardy-pract/)

[Ghoulish Delight]

Read this (http://boingboing.net/2011/02/11/ibms-watson-jeopardy.html) about it the other day, written by Bob Harris, 8-time Jeopardy winner and participant in Jeopardy Masters Tournament. I think it's a good summation of what the event "means", if anything.

[Ghoulish Delight]

So here's something I've been trying to figure out, but haven't seen anything that explains it. How does Watson understand what the categories mean. Especially the punny type categories. Does it have some definition of the kinds of puns and clue structures that Jeopardy has? Does it try to figure it out from the title? Or does it just ignore it and go purely from question content.

Judging from what we saw tonight, I'm guessing option C. Compare Watson's performance in the "APB" category to the decades category. It kicked butt in the APB category since, while the clue structure was weird, it was chock full of keywords and had very straight forward answers. Just plug in the keywords, voila, answer. Whereas the decade category, just plugging in the keywords that are in the clue would perhaps bring up a bunch of dates, but wouldn't lead directly to answering in the form of a decade (and indeed, on a few of them, the Watson display showed it wasn't even considering a decade as an answer).

So, if that's the case, I'm more impressed that it's really trying to figure out the KINDS of answers that it should be looking for, rather than just being fed the category definitions. Nifty!

And they never show enough detail on the server racks on these things. What kind of storage are they using? What about networking. I presume the servers are clustered, and using 1GbE, but is there a separate storage network. Fibre channel? iSCSI? Infiniband? (hahahaha.........okay, I'm the only one who thinks that's funny)

[lashbear]

Infiniband!!! Bwah hahahahaha !







Infiniband?

[Moonliner]

And they never show enough detail on the server racks on these things. What kind of storage are they using? What about networking. I presume the servers are clustered, and using 1GbE, but is there a separate storage network. Fibre channel? iSCSI? Infiniband? (hahahaha.........okay, I'm the only one who thinks that's funny)

Here is the best source I've seen so far on the underlying hardware (http://www.tech-forums.net/pc/f145/what-makes-ibms-watson-run-241888/).

Assuming it's not a typo, Watson has 4TB of storage (the same as my home PC) and 16TB of memory.


Watson is made up of ninety IBM POWER 750 servers, 16 Terabytes of memory, and 4 Terabytes of clustered storage. Davidian sontinued, “This is enclosed in ten racks including the servers, networking, shared disk system, and cluster controllers. These ninety POWER 750 servers have four POWER7 processors, each with eight cores. IBM Watson has a total of 2880 POWER7 cores.”


Makes sense I suppose, putting everything is RAM is the only way to get the response time you need. I bet it takes a serious amount of time to boot Watson from a cold start.

[Moonliner]

WTF! The final jeopardy question was leaked, I'm thinking the fix is in.

Commonly know as a CAPTCHA what two phrases are used here to test computer users?

http://i688.photobucket.com/albums/vv243/stevesphotobook/cap.gif

[Gn2Dlnd]

Sorry, your question must be phrased in the form of an answer.

[Cadaverous Pallor]

This has been so interesting! Watson is killing them. I think the buzzer thing just wasn't calibrated well to allow them to get in.

Watching what Watson gets wrong is so illuminating. Seems the quiz writers put in some extra effort on the Final Jeopardy question and succeeded in flummoxing Watson. Even so, his interestingly particular betting ability kept him out of trouble. The alternate answers chart on the screen including confidence in answers makes this even more fun.

It cracks me up to see the computer scientists in the audience applauding their baby.

[Alex]

Well, unless they built in an uncertainty factor (and I haven't been paying attention much) Watson should essentially always win a buzzer battle. That particular piece of technology (responding immediately to a visual signal) is pretty old and should trigger at microsecond levels.

Out of curiosity, did they eliminate video clues from the game or is Watson up to those as well?

[Ghoulish Delight]

There were no video clues in the first go around.

[Moonliner]

Well, unless they built in an uncertainty factor (and I haven't been paying attention much) Watson should essentially always win a buzzer battle. That particular piece of technology (responding immediately to a visual signal) is pretty old and should trigger at microsecond levels.



I've lost the link but one of the Jeopardy producers considered the button press issue moot because the guy that's not Ken is known for his lightning fast button times.

Obviously that is not the case. Watson is clearly winning the battle of the button press time and time again when all contestants clearly know the question long before Alex finishes reading the answer.

Still the Watson App does have to come up with the correct answer very quickly and is doing that extremely well.

[Alex]

But all Watson has to do is react to an electrical signal telling him when the buzzers are unlocked, which if it has already decided to answer should be an instant reflex.

For the humans there is a light that goes on to tell when the buzzers are unlocked. That unlocking is done by an assistant producer to tries to time it to Trebek finishing the question. So if the player waits until the light goes on then they're at an obvious huge disadvantage in terms of reflex.

But good players don't do that, they try to get a sense of the timing and guess when the light will come on so that they have started the press before it is actually possible to buzz in. If they time correctly it is theoretically possible they could slip into the window of whatever lag Watson has but that has to be an incredibly small opportunity. And the humans have a disadvantage that Watson doesn't, if the human guesses slightly wrong and buzzes in a microsecond early then they are locked out of buzzing in for a significant fraction of a second (that's why sometimes you see someone mashing the buzzer even when nobody else is trying).

So I don't see how, on those questions where Watson has decided it will answer it can consistently be beat to buzzing in.

[Moonliner]

So I don't see how, on those questions where Watson has decided it will answer it can consistently be beat to buzzing in.

Experience is showing exactly that. If Watson buzzes in he wins a disproportionate percentage of the time. You would have to penalize Watson with a delay equal to the amount of time it takes the average human to react to a stimulus in order to even things out.

Still from another point of view, the question being asked by the Watson project is can a machine answer questions as well and as fast as the best humans. In that respect I think the answer so far is yes.

[Alex]

Sure, but to the extent it is also a test of whether a machine can win at Jeopardy it has an unfair advantage due to the buzzer mechanism.

And we're not actually learning if Watson can answer more Jeopardy questions correctly than Ken Jennings. It could be that Jennings knew 58 answers to Watson's 52 but it doesn't matter if Watson gets to answer all of the 50 they have in common.

But yes, it is certainly a great demonstration of the technology, I just quibble as to what exactly it is showing.

[Kevy Baby]

... I just quibble as to what exactly it is showing.Personally, I think it should be taken as nothing more than entertainment and a unique twist on a popular show.

[Alex]

From the Jeopardy perspective, sure (though I don't expect I'd be that entertained, for the reasons mentioned, they already knew the computer could answer the questions or it wouldn't be on the show, once that's the case most suspense is gone). But from the computer science perspective I think they are important distinctions.

[Cadaverous Pallor]

Personally, I think it should be taken as nothing more than entertainment and a unique twist on a popular show.

But from the computer science perspective I think they are important distinctions.Agreed. I don't know if you're watching, Kevy, but they spent a lot of time talking about how this only one thing they can do with this technology and they hope to use it in all kinds of areas (the big example they used was health care). They're not there to demo the button pressing tech but the language parsing and decision making abilities of the machine. I was a bit disappointed in the button thing but the humans made a bit of a comeback in the last show, which was interesting. One wonders if the three of them played for days or weeks what the outcome would be.

[Kevy Baby]

We TiVod the episodes (I was out of town for a couple of days) and we just watched the first episode last night.

I was surprised to learn that Watson gets the questions as a text file: I would have thought speech recognition has evolved enough to use.

But then I started wondering - how was Watson getting the trigger to know when to push the button? But at least he has to push the same button as the players (albeit, electro-mechanically driven).

And yes, I am changing my view just a little on the show - it is a little more educational than I expected. It can be amusing to see some of the responses that Watson has been coming up with.

[Alex]

On the trigger. When the production assistant activates the buzzers after Trebek is done reading the question this turns on a light that the human players see, for Watson it sent a digital signal.

As mentioned that gives Watson an advantage since it is capable of essentially instanteous reflexes and can't (as human players risk when trying to time the activation) buzz in early causing a lockout. The other advantage is that by getting a text file Watson was probably often well on its way to an answer before the human players had even had a chance to begin comprehending what it was asking.

Like I said, it is possible that the humans knew more answers than Watson, but that is almost irrelevant if on any question that Watson also knows he is almost guaranteed winning at the buzzer.

[Kevy Baby]

So, as i see it, the two flaws in this game are:

Speech recognition should have been used (or, the text feed should have been streamed at the same pace as Alex's talking)
Warson should have to visually sense the illumination of a light (as humans must) instead of a direct link electrical signal

[Moonliner]

So, as i see it, the two flaws in this game are:

Speech recognition should have been used (or, the text feed should have been streamed at the same pace as Alex's talking)
Warson should have to visually sense the illumination of a light (as humans must) instead of a direct link electrical signal


Afraid not, the machine would still consistently win.


"Depending on the type of fiber, the neural impulse travels at speed ranging from a sluggish 2 miles per hour to, in some myelinated fibers, a breakneck 200 or more miles per hour. But even this top speed is 3 million times slower than the speed of electricity through a wire."

[Ghoulish Delight]

What Moonie said, a computer can still act faster and (more importantly) more c consistently to a light than a human. There's not a whole lot that can be done about it. The BEST they could do is introduce an artificial random delay to mimic human hesitation - but it still doesn't leave Watson susceptible to the "early trigger lockout" hazard that humans are.

[Alex]

I saw an article that asked about speech recognition and the engineers said getting a adequate recognition capability is still a decade away.

As for responding to the light that would really be just as instantaneous as responding to a signal sent directly to him (since all the signals involved are moving at the speed of light).

Oops, missed the next page with the two previous responses.

[Ghoulish Delight]

I saw an article that asked about speech recognition and the engineers said getting a adequate recognition capability is still a decade away.
[/B]I think that's a bit pessimistic. For example, Vonage recently added visual voicemail (rather, they stopped charging extra for it), and it's been remarkably accurate in its transcriptions of voicemail messages. Considering the utter crap that I saw coming out of Google Voice's visual voicemail just a couple years ago, I find it hard to believe that it will take another decade to take care of the few minor issues. Perhaps processing speed is still an issue?

But let's assume the article is right and there really is still a big gap to traverse to get fast enough voice-to-text, how about visual processing? Just have Watson read the clue off the screen. My cheap-o scanner does darn good OCR, it couldn't be that difficult to get Watson to read the very legible Jeopardy board.

[Alex]

True, I'm sure there was hyperbole in that number and probably none of the people on the Watson team are experts on the state of voice recognition. But I could see it being not so much of a problem with accurate transcription as the processing time of the transcription. 1 second of lag for visual voicemail isn't noticeable but one second of lag for Watson would have just flipped the buzz-in advantage since unlike the human players Watson wouldn't be able to read faster than Trebek talks.

OCR would be fine, but once in place, dealing with a completely standardized font it probably wouldn't be a significantly slower interface than just getting it as a text file. But that part really isn't a big deal, as Ken Jennings has said, for the best players they almost always know the answer (or have comprehended the question well enough to know they will know the answer and want to buzz in) before the buzzers are active so it all comes down to that.

To eliminate the buzz-in advantage I think what I might have done (though I haven't thought this through very much) is have run response time tests with the two human players to see what their average buzz in times were after activation for questions they knew the answer to, along with standard deviation and then programmed a random delay into sending the signal to Watson that matched that statistical distribution.

Then we'd have a true test of Jeopardy skills instead of the already known fact that a person who knows a lot of answers but always wins the buzz in will usually beat the person who knows all the answers but can't buzz in if anybody else does too. I could be Ken Jennings at Jeopardy if I always had first option to answer.

[Kevy Baby]

So basically, the best players at Jeopardy are the ones who time the button pushing the best, not necessarily the people who know the answers the best. Yes, you have to have the knowledge, but a less knowledgeable person could theoretically edge out a more knowledgeable one based on button-push timing alone.

[Moonliner]

So basically, the best players at Jeopardy are the ones who time the button pushing the best, not necessarily the people who know the answers the best. Yes, you have to have the knowledge, but a less knowledgeable person could theoretically edge out a more knowledgeable one based on button-push timing alone.

To some degree yes, but I'd bet that Ken or that other guy could handily beat me even if I had first crack at all the questions.

[Alex]

To some degree yes, but I'd bet that Ken or that other guy could handily beat me even if I had first crack at all the questions.

According to Jennings, at the level of people who make it to Jeopardy yes. He has said that at that level all the players know the vast majority of the answer before the buzzers are active and it just comes down to who can consistently win that battle.

I figure that on the average board I know 65-80% of the answers, assuming that I could resist the temptation to buzz in when I didn't, first crack should give me at least the lead going into Final Jeopardy every every time (there's still the variable of where the ones I know are distributed and who gets the daily doubles).

[Moonliner]

According to Jennings, at the level of people who make it to Jeopardy yes. He has said that at that level all the players know the vast majority of the answer before the buzzers are active and it just comes down to who can consistently win that battle.

I figure that on the average board I know 65-80% of the answers, assuming that I could resist the temptation to buzz in when I didn't, first crack should give me at least the lead going into Final Jeopardy every every time (there's still the variable of where the ones I know are distributed and who gets the daily doubles).

But where does that 20-35% you don't know fall? Wouldn't it trend towards the higher value answers?

[Alex]

But where does that 20-35% you don't know fall? Wouldn't it trend towards the higher value answers?

Not necessarily, it tends to fall into categories. If "Grunge Bands" comes up I'm only going to get the Pearl Jam question and then likely only if I answer Pearl Jam to all of them. In categories where I don't have much knowledge I can frequently say what two or three of the answers will be just from the name, the problem is I have no idea what question they match to.

But I did say "there's still the variable of where the ones I know are distributed."