Maybe we need a different representation at the upper layers, to capture higher concepts. Perhaps cross domain learning, combining learning from text for example with learning from video/images would help a lot.