17.16. ARTIFICIAL INTELLIGENCE (3.44) – TURING TEST

This is a guide for students attempting Artificial Intelligence in digital technologies achievement standard 3.44\. This guide is not official, although we intend for it to be helpful, and welcome any feedback. In order to fully cover the standard, you will also need to have done a project in one other 3.44 topic. The other project should be in either Software Engineering, Complexity and Tractability, Formal Languages, Network Protocols, or Graphics and Visual Computing. Note that the project in this guide focuses on using the Turing Test to evaluate Chatbots. Another guide exists which has a project that instead focusses on the actual algorithms and techniques used by Chatbots in an attempt to sound intelligent. Either of these are sufficient to cover a “key algorithm or technique” from the field of Artificial Intelligence. ## 17.16.1\. OVERVIEW You will need to complete a 3.44 project in two different areas of computer science. Each project needs to satisfy all bullet points in the standard. **Achieved**: [A1] **Achieved**: [A2] “describing examples of practical applications of selected areas to demonstrate the use of key algorithms and/or techniques from these areas“ **Merit**: [M1] “explaining how key algorithms or techniques are applied in selected areas” **Merit**: [M2] “explaining examples of practical applications of selected areas to demonstrate the use of key algorithms and/or techniques from these areas” **Excellence**: [E1] “discussing examples of practical applications of selected areas to demonstrate the use of key algorithms and/or techniques from these areas” **Excellence**: [E2] “evaluating the effectiveness of algorithms, techniques, or applications from selected areas” The terminology in the 3.44 standard can be challenging to understand because it applies to six different areas. The following list describes how the terminology of the standard maps onto this project. **Selected Area**: Intelligent Systems (Artificial Intelligence) **Key Problem**: Determining whether or not a computer program is able to have a conversation with a human and seem like another human. **Algorithms/ Techniques**: Turing Test **Practical Application**: Using the Turing Test to evaluate if a chatbot is able to have a human-like conversation. In summary, to satisfy the standard you might do the following: * Describe the key problem * Explain how the Turing Test can be used to evaluate a chatbot * Explain (Describe) how you evaluated a chatbot with the Turing Test, in order to demonstrate the use of the Turing Test in Artificial Intelligence. * Discuss other related applications of the Turing Test * Evaluate how effective the Turing Test is at addressing the key problem ## 17.16.2\. READING FROM THE COMPUTER SCIENCE FIELD GUIDE You should read and work through the interactives in the following sections of the CS Field Guide in order to prepare yourself for the assessed project. 10.1 - What’s the Big Picture? 10.2 - Chatterbots and the Turing Test You will be doing the activity in 10.2.6 (Run your own turing test) as the main part of your project. ## 17.16.3\. PROJECT The Turing Test is used to evaluate computer programs that are attempting to have a conversation like a human. In this project, you will carry out your own Turing test with an online chatbot, and report on the process and your findings. ### 17.16.3.1\. CARRYING OUT THE TURING TEST Choose a chatbot and carry out the Turing test with the help of a classmate, as described in activity 10.2.6\. Include both the conversations that were generated as a result of carrying out the Turing test in your report. ### 17.16.3.2\. WRITING YOUR REPORT **Achieved/ Merit** Briefly describe what the purpose of the Turing Test is. What problem in computer science does it address? Note that this should be at the start of your report, and written clearly (considering starting the sentence with “A key problem in Artificial Intelligence is…”), so that it is clear to the marker what problem the remainder of your report addresses. Explain what you did to carry out your Turing Test. Which external factors did you have to be careful of in order to ensure that the test was “fair”? Include a copy of the two conversations. Is there anything you did in your turing test that did not work very well? Were any of the questions you asked “unfair” to the chatbot? Which conversation do you think was the chatbot and which do you think was your classmate? How could you tell? Refer to some lines in your conversations (some which you think were definitely not a human). Were you correct? **Excellence** Discuss other possible applications of the Turing Test or similar tests for other programs that are supposed to be intelligent. How would you go about carrying out such a test? For example, what if you wanted to evaluate whether or not a computer played a game in a “human like” way? (As people may prefer to play against something that seems human). Discuss whether or not the Turing test is a good test of intelligence, giving reasons for your answer and using examples from your chatterbot experiments. What parts of human intelligence would the Turing test not capture? Are there are other ways of testing whether or not a computer is able to have a conversation like a human? You might like to do some additional reading for ideas for your discussion. [http://en.wikipedia.org/wiki/Turing_test#Strengths_of_the_test](http://en.wikipedia.org/wiki/Turing_test#Strengths_of_the_test) [http://en.wikipedia.org/wiki/Turing_test#Weaknesses_of_the_test](http://en.wikipedia.org/wiki/Turing_test#Weaknesses_of_the_test) ## 17.16.4\. HINTS FOR SUCCESS * Some students have used quiz show-like questions instead of trying to have conversations. It is important to try and have a conversation rather than just ask lots of unrelated questions. A conversation does best at revealing a Chatbot’s limitations. For example, this conversation isn’t very good. And infact, many of the lines could have been human responses. * judge: What is your favourite colour? * ALICE: My favorite color is green. What is yours? * judge: What do you do? * ALICE: judge, I talk to people on the web. What do you do? * judge: What is your name? * ALICE: My name is ALICE. * judge: How long have you been alive? * ALICE: 16 years. * judge: What is your favourite movie? * ALICE: My favorite movie is Casablanca. Have you seen it? * judge: What is your favourite song? * ALICE: My favorite song is Imagine. Do you know it? * judge: What is your favourite food? * ALICE: I require only electricity. * judge: What is 10 + 20 ? * ALICE: 10 times 10 = 100. * judge: What is 79 - 7 * ALICE: I would look into the web for that knowledge. Whereas this next conversation reveals the Chatbot’s limited ability to hold a conversation, as we were attempting to make it do. * judge: What is your favourite colour? * ALICE: My favorite color is green. What is yours? * judge: Mine is Purple * ALICE: purple is a very nice color. * judge: Why isn’t it your favourite colour? * ALICE: My mind does not contain an explanation for it. * judge: What does your mind contain? * ALICE: What do you think it does? * judge: I would have no idea * ALICE: How much would you pay for it? * judge: Nothing, sorry * ALICE: Really nothing? * judge: Yeah, you’re a worthless chatbot... * ALICE: Aha.. Yes. You will need to base your responses on the last response you received rather than attempting to use a list of predefined questions. * You don’t need to say the same things to the chatbot and the human. Just try and hold a conversation with both, and mostly go with whatever direction the conversation is going (although do try one or two subject changes as well to see what happens!). Just pretend you are having a casual conversation with somebody you don’t know very well. * Offensive words or topics could land you in a ton of trouble if included in your submitted report. If a chatbot says something inappropriate, censor it. Choose a classmate to work with who is likely to be sensible if you want to do well (if they do say anything offensive, don’t include it in your report). * If you have nobody to work with, consider doing the related project which focuses mostly on the chatbots and the techniques they use to try and sound intelligent. * You should be able to write up the project in about 4 pages.