Sequences: Part 1: Terminology and a simple model

12 Dec 2018

It’s been a long time since my last real post and I thought that I would share some things I’ve learned since my last post - specifically RNNs in Tensorflow (Tensorflow tensorflow, not Tensorflow Keras)

Foreword

The motivations for writing this are three-fold:

1) It seems like most resources I find online discuss Tensorflow Keras, not Tensorflow-tensorflow (aliased to tf from hereon); nothing wrong with that, but I want to write “C”, not “Python”. I’d like to know what’s happening behind the scenes for my code

2) Most tutorials online seem fishy to me. They often glossed over details that I thought were necessary especially for someone relatively new to it all - not cool for someone who is new and wants to learn these things in an “applied” manner.

3) Ever since joining my current company, I’ve become focused on the idea of “production” code. Most tutorials I see are roughly hacked together and look like the dogs breakfast. Also, I’ve heard (anecdotally) that converting your keras code into tensorflow serving can be a pain in the ass

Side note: there are many excellent blog posts that teach you how to derive the gradients for RNNs and the math behind them. I suggest looking towards those if that’s what you want

Plan

I’m not sure how many tutorials I plan to have - I’d like to cover

1) basic RNNs

2) Temporal Convolutional Networks

3) Seq2Seq models

4) Techniques and different losses (for example, things like Attention, or a beam search decoder)

Terminology

Seq2Seq: A problem class consisting of 2 models where one encodes and the other decodes

RNNs and LSTMs: Models that can be utilized for different problem classes where there is a time-component.

Side note: Given how much earlier RNNs and LSTMs came before Seq2Seq models, I had initially thought that traditional RNNs cannot predict sequences (this is not the case). Perhaps this was obvious to people (I didn’t read the original paper) but this was not obvious to me.

Simple Model

Setup

0) Check back often, as after the Tf Pipeline series is finished, I’ll work on the Production series which will aims to productionize all of this

1) Look at my blog for the requirements.txt file to run the setup and such

[ Sequences  Production  Tutorial  ]