Intro

Here, I just scratch the surface of NLP, by commenting my colab notebook which builds the backend of a sentiment movie review app from the ground up exploiting Keras’ TextVectorization.

You are expected to be familiar with the following concepts and tools:

• Basics of Machine learning and data manipulation: did you try the Titanic dataset?
• Basics of Keras API: you got at least on neural network with Keras tutorial

I will try to provide crash-course on word vectors and text vectorization.

Turning Words into numbers: Text vectorization

A computer can naturally manipulate and operate on numeric representations of entities but not on text or strings. Understanding and managing effectively at least one way of translating text into numbers-useful-for Machine Learning, should be in the tool belt of any ML practitioner.

Data representation: the need for good word vectors

Think of performing a regression: you want to associate a numeric output (y) to numeric inputs (X). The definition of (X) is easy when you have tabular data, i.e. the kind of data which is stored in an Excel spreadsheet or in a database. An example could be predicting the value of a used car (y) depending on its age (x1), kilometers (x2) and horsepower (x3) ¹.

To improve your predictions, you may also want to include a non-numeric attribute, as the car market segment (Compact, Suv, Wagon)? In this case, the information is available as a string but numeric representations are well-established and easy to create, as one-hot-encoding, which associates to each car (c) a 3D vector (i,j,k) whose entries are 1 if (c) posses the attribute and 0 otherwise. So, if (c) is a compact one, its “market segment” attribure will be (1,0,0) while if it is a station wagon it’ll be (0,0,1).

Now think of having also image data. A lot of information there and the technology to represent an image using numbers has long been there. A quite intuitive representation for a grayscale image is a 2D matrix of pixel with values ranging from 0 (white) to (255) black. The problem with image data is that for long time it has been difficult to build a model able to make sense of X to create a good (X)->(y) relationship. Back to our value of used car example, a simple statistical model is not able to extract useful information (color, size, design) from a car image.

You may well be seeing where we are going. What happens when want to exploit unstructured textual data?

If we had to start from scratch, we may try to approach the problem by constructing one-hot-encoded word vectors.

Again, in our example, we may have a series of reviews associated with the car model, let’s for instance take the pro & cons of Fiat 500 Convertible for a concrete example of real-world data:

Nimble size for easy parking, Fairly comfortable ride quality for a car of this size, Lots of personality for the price, Disappointing fuel economy given the slow acceleration, Poor rear visibility with the convertible’s top lowered.

Notice how descriptive (and difficult to preprocess) textual data may be. We have 30 different vocabulary words here just for a single piece of text from the data. Even if we could build those one-hot vectors there are at least 2 very clear drawbacks:

1. the number of “numeric” values would be very high. In other words this representation has a very high cardinality: this is a size and computation problem².
2. this representation is very sparse and your model can have a really hard time in finding good

A possible solution to this is using Bag of Words model, another one -which I usually prefer- is via words embedding³.

Words embedding: training from scratch

Writing… (this is connected with the original notebook, we just use movie-reviews toy problem).

Words embedding: re-use

Writing…

Remember that word embedding may shift with time!

• Dynamic Word Embeddings