Ideas from Elman's Finding Structure in Time (1990)
Simple Recurrent Networks
- An extension of simpler feedforward networks to incorporate time
- Hidden layer activations are copied to a context layer and fed
back into the network on the next time step
- Using this recurrence, the network can develop a memory of the recent past
Finger tapping experiments
- 1 2 3 4 5 1 2 3 4 5 ...
- 1 2 3 4 5 1 3 5 2 4 ...
- 1 2 3 1 2 4 1 2 5 5 ...
- Human subjects have varying degrees of difficulty repeating these
- Simple recurrent networks show similar results (exposures needed
to learn by sequence: 250, 1500, 30000)
Discovering the structure in letter sequences
ba, dii, guuu
Discovering the notion of "word"
Discovering lexical classes from word order
Types, tokens, and structured representations
Summary of Elman's results
- SRNs are trained to predict symbolic sequences
- SRNs create a tight coupling between input and output
- Self-organized internal representations emerge
- Differential error rates indicate that SRNs "recognize"
sub-structure of sequences
- Errors are not random but reflect structured representations
- This is a first step towards developing concepts
Applying simple recurrent networks to robots
- The real world exists in time
- Robots experience the world through their sensors
- Robots must learn to recognize key events in this stream of
- Unlike the controlled experiments done by Elman, a robot may
experience long periods of essentially static input
- In this case, the tight coupling of inputs and outputs in an SRN
can lead to catastrophic forgetting
- Our proposed solution could be described as a bipartite systems, one
coupled with current experiences, while the other attempts to
categorize with respect to all experiences
- Results so far suggest that each part individiually has a large
weakness, but when they are trained together they can learn patterns
(and therefore concepts) that an SRN cannot