Discussion & links to various implementations: https://www.reddit.com/r/MachineLearning/comments/eg1wr3/ref...

The demonstration on images is underwhelming at best. It is only marginally better than extending vertically the bottom row of pixels.

There is no argument for why the LSH would work well, especially at the beginning of training. As the weights are initially random, bucket assignment would be random as well. If predicting at position A requires info from position B, but they are not in the same bucket, there will be no gradient to get the query embedding of A closer to the key embedding of B. The reversible layer trick is neat though.

One neat trick is that you can extend GPT-2 117M's context window from 1024 up to 30k on a TPU, since TPUs can allocate up to 300GB of memory for backprop. https://twitter.com/gwern/status/1218001309435072513

It's not quite 1M words, but a 30k context window is big enough for e.g. most midi songs.

This seems like a big deal. An asymptotic reduction in the resource explosion created by larger attention windows should allow the development of substantially more complex models here.

Vowpal Wabbit has been doing this 'hashing trick' since the 200s.

It also the feature interaction, which are the same thing as a layer in transformers (all against all matrix).

So it seems like they are still catching up to where John Langford and crew were over a decade ago.

And, the vowpal wabbit approach is extremely fast to train because it's only doing stochastic gradient descent on a linear function - linear regression. Transformers are much slower to train.

EDIT: Downvoters, please see my last leaf to see why they're effectively the same. The guy responding here seems unfamiliar with all the functionality of vowpal wabbit.

I wonder if this could be used for the Wikipedia compression challenge?

How does accuracy compare in Nlp tasks vs XLnet? If we can have XLnet accuracy and fast inference on a single gpu, that would be revolutionary!

This looks like building blocks from cryptography inspiring ML

Would it be reasonable to add something to the tittle so it's clear it has nothing to do with electronics? Maybe it's just me.

so many smart people and still using fuzzy PNG instead of SVG