2018-12-17: CoQA Challenge: Machine Reading Competition Recent Result
Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters. All of her sisters were cute and fluffy, like Cotton. But she was the only white one in the bunch. The rest of her sisters were all orange with beautiful white tiger stripes like Cotton's mommy. Being different made Cotton quite sad. She often wished she looked like the rest of her family. So one day, when Cotton found a can of the old farmer's orange paint, she used it to paint herself like them. When her mommy and sisters found her they started laughing. "What are you doing, Cotton?!" "I only wanted to be more like you". Cotton's mommy rubbed her face on Cotton's and said "Oh Cotton, but your fur is so pretty and special, like you. We would never want you to be any other way". And with that, Cotton's mommy picked her up and dropped her into a big bucket of water. When Cotton came out she was herself again. Her sisters licked her face until Cotton's fur was all all dry. "Don't ever do that again, Cotton!" they all cried. "Next time you might mess up that pretty white fur of yours and we wouldn't want that!" Then Cotton thought, "I change my mind. I like being special".
This reads like a picture book story, so you can see what kind of text current machine reading can achieve. The sample questions and their answers are
Q What color was Cotton?A white || a little white kitten named Cotton A white || white kitten named Cotton A white || white A white || white kitten named Cotton.Q Where did she live?A in a barn || in a barn near a farm house, there lived a little white kitten A in a barn || in a barn near a farm house, there lived a little white kitten named Cotton A in a barn || in a barn A in a barn near || in a barn near a farm house, there lived a little white kitten named Cotton.Q Did she live alone?A no || Cotton wasn't alone A no || But Cotton wasn't alone A No || wasn't alone A no || But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters.Note that there could be multiple answers because they are given based on different sentences quoted from the story. So these sentences are used as the explanations or justifications of the answers.
Up until November 2018, the best model is an ensemble model called SDNet developed by Microsoft with an overall accuracy of about 79%. In December 2018, iFlyTek and HIT (Harbin Institute of Technology) beats them and achieves an overall accuracy of about 80% using a single model. iFlyTek is a Chinese IT company and HIT is a research institute in China. The SDNet model and iFlyTek model both adopt Google's BERT module. The Stanford NLP group is at #8 with an accuracy of 65%. AllenAI is at #4 following Microsoft (single model) with an accuracy of 75%. This represents the best performance of QA systems nowadays. The SDNet system is described in a paper on arXiv.
For the most recent result, please see the front page of the competition website.
Below is copied directly from the competition website.
The unique features of CoQA include 1) the questions are conversational; 2) the answers can be free-form text; 3) each answer also comes with an evidence subsequence highlighted in the passage; and 4) the passages are collected from seven diverse domains. CoQA has a lot of challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning.
Jian Wu
Comments
Post a Comment