Text2Action
The contents in Github Repository includes the implementation of the Text2Action network.
Text2Action: Generative Adversarial Synthesis from Language to Action
- Article: Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh, "Text2Action: Generative Adversarial Synthesis from Language to Action," in Proc. of the IEEE International Conference on Robotics and Automation (ICRA), May 2018. [Video | arXiv preprint]
- Abstract: In this paper, we propose a generative model which learns the relationship between language and human action in order to generate a human action sequence given a sentence describing human behavior. The proposed generative model is a generative adversarial network (GAN), which is based on the sequence to sequence (SEQ2SEQ) model. Using the proposed generative network, we can synthesize various actions for a robot or a virtual agent using a text encoder recurrent neural network (RNN) and an action decoder RNN. The proposed generative network is trained from 29,770 pairs of actions and sentence annotations extracted from MSR-Video-to- Text (MSR-VTT), a large-scale video dataset. We demonstrate that the network can generate human-like actions which can be transferred to a Baxter robot, such that the robot performs an action based on a provided sentence. Results show that the proposed generative network correctly models the relationship between language and action and can generate a diverse set of actions from the same sentence.
- Bibtex entry:
@inproceedings {ahn:Text2Action:icra18,
author = {Hyemin Ahn and Timothy Ha and Yunho Choi and Hwiyeon Yoo and Songhwai Oh},
title = {Text2Action: Generative Adversarial Synthesis from Language to Action},
bocktitle = {Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
month = {May},
year = {2018}
}
DEMO
An example is provided in Github Repository (https://github.com/hiddenmaze/Text2Action)
Download
This software is made available for free for non-commercial use. The software must not be modified or distributed without prior permission of the author.
Current version: 0.1, March, 2018