Basic Operations on Tensors

This article is part two of the Linear Algebra series. The first part gave a basic introduction to the fundamentals of Linear Algebra: scalars, vectors, matrices and tensors. The article also discussed scalars being 0^th order tensors, vectors being 1^st order tensors and matrices being 2^nd order tensors. In this article, we’ll see the basic operations that can be performed on tensors. The following tensor operations are discussed.

1. Addition
2. Broadcasting
3. Multiplication (including dot product and Hadamard Product)
4. Transpose
5. Inverse

Let’s get started.

Addition

A normal matrix addition involves element wise addition. Let us consider two matrices A and B and their sum resulting in a new matrix C.

A_{ij} + B_{ij} = C_{ij}

For addition of two matrices, the dimensions of the matrices must match. The final resulting matrix would also be of the same dimension.

Example:

If A = \begin{pmatrix} 1 & 2\\ 3 & 4 \end{pmatrix}, B = \begin{pmatrix} 5 & 6\\ 7 & 8 \end{pmatrix}, then the addition of the matrices would go like this:

\begin{pmatrix} 1 & 2\\ 3 & 4 \end{pmatrix} + \begin{pmatrix} 5 & 6\\ 7 & 8 \end{pmatrix} = \begin{pmatrix} 6 & 8\\ 10 & 12 \end{pmatrix}

Broadcasting

This is a special type of tensor addition operation where a matrix and a vector is added. Earlier, we saw that the addition requires two tensors to have same dimension for element wise sum. However, we already know that vector is a 1-D array while a matrix is 2-D array. Therefore broadcasting is basically a programmatic approach of adding a matrix and a vector. The vector is automatically replicated to match with the dimension of the matrix it is going to be added to.

Here’s the mathematical notation of broadcasting: A_{ij} + v_{j} = C_{ij}, where A_{ij} is a matrix and v_{j} is a vector. The dimension of the vector, however, should match either the number of rows or the number of columns of the matrix it is going to be added to. Only this way, the vector can replicate itself for addition.

NumPy and MATLAB both support broadcasting.

Example:

Let us suppose, A = \begin{pmatrix} 1 & 3 & 5\\ 2 & 4 & 6 \end{pmatrix}, v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}

Although the dimensions do not match, the vector is replicated as a part of broadcasting and the following result is produced after addition:

C = \begin{pmatrix} 2 & 4 & 6\\ 4 & 6 & 8 \end{pmatrix}

Multiplication

Multiplication of a matrix A of dimension m*n and a matrix B of dimension n*q is given by:

C_{ij} = \sum_{k}A_{ik}B_{kj}

For multiplication, the number of columns of the first matrix must match with the number of rows of the second matrix.

Example:

Let’s suppose two matrices, A = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix} and B = \begin{pmatrix} 1 & 2 \\ 3 & 1\\ 2 & 3 \end{pmatrix}

C_{ij} = \begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix} * \begin{pmatrix} 1 & 2 \\ 3 & 1\\ 2 & 3 \end{pmatrix} = \begin{pmatrix} 1*1+2*3+3*2 & 1*2+2*1+3*3\\ 4*1+5*3+6*2 & 4*2+5*1+6*3 \end{pmatrix} C_{ij} = \begin{pmatrix} 13 & 13\\ 31 & 31 \end{pmatrix}

Multiplication of a matrix and a vector (column vector) is also possible until the dimension conditions match.

Hadamard Product

The Hadamard product involves element wise multiplication. Multiplying matrices must be of same dimensions.

C_{ij} = A_{ij} \odot B_{ij}

Example:

\begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{pmatrix} \odot \begin{pmatrix} 2 & 2 & 3\\ 4 & 2 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 4 & 9\\ 16 & 10 & 6 \end{pmatrix}

Dot Product

A dot product of two vectors results into a scalar. The dot product between vectors \overset{\rightarrow}{x} and \overset{\rightarrow}{y} is defined as (in vector notation):

\overset{\rightarrow}{x}.\overset{\rightarrow}{y} = a \sum_{i}x_{i}y_{i} = a

where, a is scalar.

Example:

Let’s suppose \overset{\rightarrow}{x} = \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix} and \overset{\rightarrow}{y} = \begin{bmatrix} 2 & 3 & 1 & 4 \end{bmatrix}

= \begin{bmatrix} 1 & 2 & 3 & 4 \end{bmatrix} . \begin{bmatrix} 2 & 3 & 1 & 4 \end{bmatrix} = \begin{bmatrix} 1*2+2*3+3*1+4*4\end{bmatrix} = 27

In matrix notation, the dot product is written as:

x^{T}*y = a or x*y^{T} = a

where the superscript T denotes the transpose operation. The transpose makes sure that the two vectors are compatible for multiplication. The transpose operation is performed in any one of the multiplying vectors.

Transpose

The transpose of a matrix A which denoted by B = A^{T}, is given by B_{ji} = A_{ij}.

Example:

Let’s suppose a matrix A = \begin{pmatrix} 1 & 2 & 3\\ 3 & 4 & 5 \end{pmatrix}. The transpose A^{T} is:

B = A^{T} = \begin{pmatrix} 1 & 3\\ 2 & 4\\ 3 & 5 \end{pmatrix}

Inverse

A inverse matrix (A^{-1}) is a matrix which when multiplied with the original matrix (A) gives an identity matrix (I). An identity matrix is a special kind of square matrix that has ones in the diagonal with all other elements as zero.

I = A^{-1}A = AA^{-1}

An identity matrix I of dimension 3*3 is defined as:

I = \begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix}

Not all square matrices will have an inverse. In the case of non-square matrices, we have Moore-Penrose pseudo inverse for the generalization of the matrix inverse.

These are the basic operations commonly performed on the tensors. In the next post I’ll come up with more detailed ideas related to Linear algebra. See you in the next one.