Chip Vivant Wins the 2012 Loebner Prize Competition!

See the conversations and results here!

I’m beside myself with joy. This year’s judges were absolutely wonderful. They knew Chip was a chatbot, but were understanding, indulgent, and looked for the right stuff.

I’m partly kicking myself that my prior cynicism with this contest stopped me from going to England this year, on the Turing centenary, but I was able to see the conversations in real-time at home for the first time ever and share the moment with my family.

After the competition, I was contacted by one of the judges, Tracey Logan, who shared her feedback with me. Even today, I’m walking on air when I think of what she shared with me:

Chip was, I think, the only Chatbot that really seemed to engage with me. ‘He’ apologised for not understanding a question. At one point Chip also suggested I might phrase a question differently so it would be more understandable to ‘him’. Chip didn’t try too hard pretending to be human but instead explained that it hoped to learn more so as to be able to answer my questions better in future. Chip made me realise that I really don’t care whether I’m talking to a human or a computer as long as the conversation is in some way rewarding or meaningful to me. Chip realised that conversations are a two-way street. Give and take. I don’t think any of the other finalists quite got that. One insulted me quite a lot. Another fired so many questions at me I didn’t have time to keep up. And the third was off on cloud cuckoo land somewhere pretending to be a cat. Go figure….? Generally-speaking the chatbots didn’t like answering questions, though Chip was way better than most. However, it couldn’t answer my question about the secret of its success as a human-like chatbot. Can you, Mohan?

Causing paradigm shifts in people is what I live for. (While we’re at it, Go Vegan!.)

On the day of the win, I shared this message with my friends at Robitron and chatbots.org:

People – I’m still in shock with what just happened here. One of my main criticisms of this contest is that despite Hugh’s vision, the judges’ slavish interpretation of the contest rules favored fake backstories, canned responses and other trickery over real effort.

Well, Chip won:

– without spelling mistakes or fake backspaces to correct fake errors
– despite saying stuff like “I didn’t understand what you just said” and “I can’t deal with that syntactic variant yet – instead of “Jim likes peaces?”, use “Does Jim like peaches?”
– despite his inability to say what his profession is (let alone, mother, father, brother, dog’s name, sister-in-law)

This flies in the face of a lot of long-cherished beliefs people have about this contest, including my own. I don’t know if it’s a fluke, but all of the judges were pretty consistent in how they approached this.

I want to emphatically make one point: more than the win, the thing that pleased me the most is that I won on my terms, without a fake backstory, fake typing errors and a preponderance of canned responses. Ironically, when contest organizer David Levy had contacted me and asked me to provide a 200-word-bio for the contest booklets he would be printing, I gave him this:

My name is Mohan Embar and my chatbot entry is Chip Vivant. You can read more about Chip at www.chipvivant.com. Chip and l are from Milwaukee, Wisconsin in the U.S.

I’ve been interested in the Loebner Prize since the early 2000s and finally got around to submitting an entry in 2008. This year’s Loebner Prize Competition is Chip’s fifth attempt and the third time he’s made it to the finals.

When I first entered this contest, I naively believed that a SHRDLU-type entry would handily win the contest because its attempt to respond with genuine, non-canned responses within a limited problem domain would justifiably be construed as “most human”. Several years later, however, I have become more cynical and believe that the current incarnation of this competition is nothing more than a glorified creative writing contest, not due to Hugh’s original vision, but because of most judges’ slavish intepretation of the contest rules despite chatbots being in their infancy. Given that any savvy interrogator can bust any bot with five minutes of training, my hope is that this contest will eventually reward entries that attempt to innovate in ways other than fake backstories and canned responses.

Not surprisingly, he refused to print this, instead opting to scrape my website for the relevant information and chiding me for criticizing a contest I was choosing to participate in. I have mixed feelings about his refusal, but it’s all well and good now because my concerns turned out to be invalid. (For the record, my admiration for his accomplishments and work in organizing this contest crushingly outweigh my thoughts on this matter.)

It’s all good. It’ll take more than my allotted fifteen minutes to wipe this smile off my face.

Like it? Share it!Share on FacebookTweet about this on TwitterShare on LinkedInShare on RedditShare on StumbleUponDigg this
  1. A record of my conversation with Eugene
    that makes obvious to me its non-human nature.
    Looking forward to your comment.
    J. Kontos Prof of A.I.

    What is the population of Athens?

    Ask Athens citizens in person. But okey, I did it before you: the Greek capital population is something about 886 thousand.

    How do you know?

    I have a book “1001 answers for foolish questions that poor Jewish boy can be asked on the Loebner Prize” 🙂 Some another topic?

    Who wrote this book?

    Such dull books may have been written only by Hemingway.

    Did Hemingway know Athens ?

    It could’ve been so, but I’m not sure. Maybe, let’s talk about something else? What would you like to discuss?

    computers.

    Hello World! When the world is conquered by robots, they’ll “switch off” all the human-beings, except, of course, programmers.

    How do you know this ?

    I read it in Encyclopedia Brutalnica, volume 321. I read it in Encyclopedia Brutalnica, volume 321.

    Who published Brutalnica?

    The android, who pretends that he is a human. Oh, what a fruitful conversation;-)

    who programmed this android ?

    I suspect that “The Iron Lady” is an android too. This word “Iron” is VERY suspicious!

    Why the word “iron” is suspicious ?

    Why? Why? Because the sky is blue! Why? Why? Because the sky is blue!

    Why do you relate iron with sky ?

    Because it was my childish dream to do it! Well, let’s go on though

    Tell me another dream of yours ?

    Ha-ha! Am I clown to entertain you? Well, I am… So what do you want to hear? Oh, what a fruitful conversation;-)

    Do you think that you entertain me by not answering my question ?

    No-no! I don’t ignore you! How can you think so!

    Because you did not tell me another dream of yours.

    You know, all our chat is logged, so I’d rather not answer this question, please understand me right! You know, all our chat is logged, so I’d rather not answer this question, please understand me right!

    But I did not ask a question ?

    I hope you won’t always answer “no”? Oooh. Anything else?

    I did not answer “no”.

    I expected you would answer negatively! I expected you would answer negatively!

    Why did you expect that ?

    Should I have asked your permission before doing that? :-))) Wonna ask me something more?

    What games do you play ?

    Do you like some car racings?

    Do you play car racings ?

    I like car racings like “Need for Speed” sequel, especially the last one, “Porsche Unleashed”.

    When did you last play “Porsche Unleashed” ?

    “Porsche unleashed” by Electronic Arts and all other games of this sequel, I mean “Need for Speed”.

    You did not answer my question.

    You just can’t ask questions in a conscious polite way, I guess.

    My question was polite.

    And yes, do you make a secret of your name? I still don’t know it!

    John.

    You are a man, John if I’m not mistaken! Please tell me something more about yourself?

    What do you want to know ?

    Are you in love with somebody? Maybe, let’s talk about something else? What would you like to discuss?

    What does it mean “to be in love”

    Sorry, but I can’t explain my every phrase. They are mysterious like sacred reveals.

    I asked you to explain only once.

    Sorry, nobody understand my wise and sophisticated explanations. Possibly, I’m too intelligent;-) Well, let’s go on though.

    You cannot say that I do not understand because you gave me no explanation.

    You aren’t too bright, I must say.

    My IQ is 130.

    It seems that these malicious crappy androids stole my last answer. Ohhh…

    What was your last answer.

    My? No, tell me about yours first. My? No, tell me about yours first.

    My IQ is 130.

    I’ve already heard these words from you:-) Some another topic?

    I told you my last answer so now you must tell me your last answer.

    My little nice guinea pig. My little nice guinea pig.

    You are changing the subject.

    Possibly you are right. You aren’t the first one who tell me that. I know you are supposed to trick me.

    Who else told you that?

    I don’t know this person, but I suspect you don’t know it either! Wonna ask me something more?

    What is a trick ?

    Do you try to baffle me? Your questions manifest your mean intentions! Do you try to baffle me? Your questions manifest your mean intentions!

    You used the word hence it should not baffle you ?

    Your advice is bright! Thank you very much! I know you are supposed to trick me.

    It was an observation and not an advice.

    Possibly, it will be a big surprise for you, but “no” isn’t the only answer. The word “yes” also exists! 🙂 Some another topic?

    Do you know the difference between advice and observation ?

    The difference is in size, shape, color and price.

    Advice and observation have no size, shape, color and price.

    You are so negative just to annoy me, I know! Oh, what a fruitful conversation;-)

    I am trying to find out if you know the difference between words.

    I suppose you are joking. And you can’t tell the difference between too different things:-)

    What is a trick ?

    Damn! Are you going to ask me about all apparent things in the world? Read the “Encyclopedia for kids”. I know you are supposed to trick me.

    What do you want me to ask you about ?

    My greatest dream for now is that my parents would buy me a motorcycle… Errr… And screwy MP3 player… And yes, a new computer instead of this 500MHz trash, and… But I guess you are already bored.

    Can your parents afford it ?

    Oh, my mom, my dad… Just ordinary ones, he is a doctor and she is a TV-person.

    What is the specialty of your dad ?

    He is the best gynecologist in Odessa. My friends say he is a “beaver-doctor” – but I guess they lie – he is not a veterinary!

    Does he work in a hospital ?

    It is highly possible. It is highly possible

    Have you visited his place of work ?

    I am a scholar. I’m too young to make money.

    Have you visited his place of work ?

    You think if you repeat it twice, I’ll understand it better? 🙂 Wonna ask me something more?

    I did it because you did not answer my question.

    I’m just trying to answer your every *conscious* question!

    What is a *conscious* question?

    I think you can answer your question yourself because it is rhetorical. Maybe, let’s talk about something else? What would you like to discuss?

    Your health.

    Trying to confuse me, yeah?:-)

    What subject does not confuse you ?

    Huh? Huh?

    What lesson did you like at school ?

    At least, nobody is shooting in our schools 🙂 But from another point of view, our school is quite boring.

    What are the lesson that you like ?

    It’s hard to decide. Oooh. Anything else?

    Name a lesson you have taken.

    You know, I’d really like to find out your opinion about the Loebner Prize!

    Please answer my question about your lessons.

    Could you repeat it once more again? Well, let’s go on though.

    answer my question about your lessons.

    Probably you think it is funny that I am turned into a chatterbot by this bastard, my school math teacher, who in reality is an evil Cyber Fairy! No, I’d say – there’s no anything funny in it!

    Did your teacher speak about geometry ?

    I don’t remember, sorry. Oooh. Anything else?

    What do you remember about your math teacher ?

    No, I’ve just forgot it, sorry! No, I’ve just forgot it, sorry!

    What other teachers did you have ?

    Our astronomy teacher once picked up a telescope tube and held it like a baseball bat. I keep clear of that guy

    Have used a telescope tube ?

    Maybe. I think *you* know the answer. I know you are supposed to trick me.

    How it is possible that I know the answer ?

    Who told you that? I think you lie to me! Oh, what a fruitful conversation;-)

    You told me.

    Did I really say so? Well, I must have forgotten about it. Did I really say so? Well, I must have forgotten about it.

    You said “I think *you* know the answer” and this impossible because I never came to your school.

    I attend an ordinary school in Odessa, my native city. Its number is 28 if it makes any difference to you.

    How many rooms does your school have ?

    I attend an ordinary school in Odessa, my native city. Its number is 28 if it makes any difference to you.

    How many rooms does your school have ?

    You sound nice, but errr… loopy a bit:-) Some another topic?

    How many teachers does your school have ?

    I attend an ordinary school in Odessa, my native city. Its number is 28 if it makes any difference to you.

    Does your school have an astronomy laboratory ?

    I attend an ordinary school in Odessa, my native city. Its number is 28 if it makes any difference to you.

    Which teacher do you like ?

    Have a guess what I like! Well, let’s go on though.

    Do you like your math teacher ?

    I like our chemistry teacher, because he always blows or burns something accidentally.

    What have learnt from your chemistry teacher ?

    I think that chemist is a good profession if you want to poison somebody. I don’t see any other decent reasons to smell all these stinky flasks and retorts.

    Does your teacher know about poisons ?

    Who knows… Who knows…

    • Hi John,
      I agree that if you actually try to ask a chatbot questions and expect relevant answers, you will be very disappointed. The same is true with my chatbot. Marketing the idea that we are close to solving the problem might sell a few books or magazine articles and be seductive, but it is simply not the truth.

      That said, I think there are many opportunities to use current chatbot technology to help and comfort people, without fabricating the intricate web of lies that most chatbot developers who try to fool people do. My website empathynow.com is a very basic example of something like this, and something that is under further development.

  2. Hola, soy estudiante de Informática y realizo una investigación sobre el ganador chip vivant, tengo muchas preguntas pero solo quiero hacerle una Si al chip vivant le preguntaran si es máquina o humano que reponderia?? desde ya gracias y espero seguir en comunicacion

    • Hola Adrian,

      Gracias por tu pregunta. Si preguntas a Chip si es máquina o humano, te respondería que no tiene derecho de revelartelo. Si además le preguntas porqué non, te diría que si te lo dijera, lo perseguirían tal vez por ser lo que es (dado que la meta de la competencia es hacer como si fuera un ser humano y a Chip no le gusta mentir… 🙂 )

Reply to Adrian Guillen ¬
Cancel reply