Deep Learning Frameworks and TensorFlow

deep-learning
tensorflow
frameworks
gradient-tape
How to choose a deep learning framework, and the basic structure of a TensorFlow program with variables, GradientTape, and optimizers.
Published

Aug 6, 2026

You have learned to implement deep learning algorithms more or less from scratch using Python and NumPy, and that was worth doing because it shows what these algorithms are really doing. But unless you implement more complex models, such as convolutional or recurrent neural networks, or start to implement very large models, doing everything yourself from scratch becomes increasingly impractical. Fortunately, there are now many good deep learning software frameworks that help you implement these models.

Choosing a Deep Learning Framework

To make an analogy, hopefully you understand how matrix multiplication works and could code up a function that multiplies two matrices yourself. But as you build very large applications, you probably do not want to implement your own matrix multiplication function; you want to call a numerical linear algebra library that does it more efficiently for you. It still helps that you understand how multiplying two matrices works. Deep learning has now matured to that point; it is usually more practical and more efficient to build with one of the deep learning frameworks.

There are many frameworks today that make it easy to implement neural networks. The leading ones are PyTorch, which dominates research and an increasing share of production work, TensorFlow, which remains common in industry deployment pipelines, Keras, which is now a multi-backend interface that can run on top of either of those, and JAX, which pairs a NumPy-style interface with automatic differentiation and compilation and is popular in research. Sitting one level above them is Hugging Face Transformers, which is not a framework in this sense but a library of pretrained models built mostly on PyTorch; many modern projects start from a pretrained model there rather than building a network from scratch. Each of the leading frameworks has a dedicated user and developer community, and each is a credible choice for some subset of applications. Plenty of articles compare them, and because these frameworks often evolve and get better month to month, you can do a few internet searches yourself if you want the current pros and cons. Rather than strongly endorsing any single framework, here are the criteria worth using to choose one.

  1. Ease of programming. That means both developing the neural network and iterating on it, as well as deploying it for production, for actual use by thousands or millions or hundreds of millions of users, depending on what you are trying to do.
  2. Running speed. Especially when training on large datasets, some frameworks let you run and train your network more efficiently than others.
  3. Whether the framework is truly open. This one is not talked about as often, but it matters. For a framework to be truly open, it needs not only to be open source but to have good governance as well. Some companies in the software industry have a history of open sourcing software while maintaining single-corporation control of it, and then, over some number of years as people come to depend on it, gradually closing off what was open or moving functionality into proprietary cloud services. So it is worth paying attention to how much you trust that a framework will remain open source for a long time, rather than being under the control of a single company that may choose to close it off in the future.

Beyond that, depending on your preferences of language, whether you prefer Python, Java, C++, or something else, and depending on your application, computer vision, natural language processing, online advertising, or something else, multiple frameworks can be a good choice. By providing a higher level of abstraction than a numerical linear algebra library, any of these frameworks can make you more efficient as you develop machine learning applications.

Review Questions

1. What three criteria are recommended for choosing a deep learning framework?

First, ease of programming, both for developing and iterating on the network and for deploying it to production at scale. Second, running speed, especially for training on large datasets. Third, whether the framework is truly open, meaning open source with good governance, so you can trust it will remain open rather than being gradually closed off or folded into proprietary cloud services by the single company controlling it.

1. If frameworks do the work anyway, why was it worth learning to implement neural networks from scratch in NumPy?

For the same reason it helps to understand how matrix multiplication works even though you call a linear algebra library in practice. Understanding what the algorithms are really doing lets you use the higher-level abstraction effectively; the framework replaces the labor, not the understanding.

1. Which of the following are some recommended criteria to choose a deep learning framework?

  1. It must run exclusively on cloud services, to ensure its robustness.

  2. It must use Python as the primary language.

  3. Running speed.

  4. It must be implemented in C to be faster.

c. Running speed is a major factor, especially when training on large datasets. The other recommended criteria from this section are ease of programming and whether the framework is truly open; nothing requires a particular language or implementation, and running exclusively on cloud services would work against openness.

1. If a project is open source, that is a guarantee it will remain open source in the long run and will never be modified to benefit only one company. True or False?

  1. True

  2. False

b. False. Being open source today is not a guarantee about the future. To trust that a project will remain open in the long run, it must also have a good governance body, since a project under the control of a single company can gradually be closed off or folded into proprietary services as people come to depend on it.

Basic Structure of a TensorFlow Program

One of these frameworks is TensorFlow. This section steps through the basic structure of a TensorFlow program, starting from a motivating problem. Suppose you have a cost function \(J\) to minimize, in this example the highly simple

\[ J(w) = w^2 - 10w + 25 \]

You might notice this is actually \((w - 5)^2\); expanding out that quadratic gives the expression above, so the value of \(w\) that minimizes it is \(w = 5\). But say we did not know that and just had the function. A program with a very similar structure can be used to train real neural networks, where the cost \(J(w, b)\) can be a complicated function of all the parameters of the network, and TensorFlow automatically tries to find values that minimize it.

To start up TensorFlow, the idiomatic imports that pretty much everyone types are

import numpy as np
import tensorflow as tf

Next, define the parameter \(w\). In TensorFlow you use tf.Variable to signify that this is a variable, initialize it to 0, and give it the type tf.float32, a TensorFlow floating point number. Then define the optimization algorithm you are going to use, in this case the Adam optimization algorithm with the learning rate set to 0.1.

w = tf.Variable(0, dtype=tf.float32)
optimizer = tf.keras.optimizers.Adam(0.1)

The great thing about TensorFlow is that you only have to implement forward prop, that is, you only have to write the code that computes the value of the cost function, and TensorFlow can figure out how to do the backprop, the gradient computation. One way to do this is with a gradient tape. The intuition behind the name is an analogy to old-school cassette tapes. tf.GradientTape records the sequence of operations as you compute the cost in the forward prop step, and when you play the tape backwards, it revisits the operations in reverse order and, along the way, computes backprop and the gradients.

A single training step, one iteration of training, looks like this. You define which variables are trainable (a list containing only w here), compute the gradients with tape.gradient, and then use the optimizer to apply them. The built-in Python zip function pairs up the corresponding elements of the two lists, gradients with trainable variables.

def train_step():
    with tf.GradientTape() as tape:
        cost = w ** 2 - 10 * w + 25
    trainable_variables = [w]
    grads = tape.gradient(cost, trainable_variables)
    optimizer.apply_gradients(zip(grads, trainable_variables))

print(w)
<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0>

We have not actually run train_step yet, so \(w\) is still the value 0 that we initialized it to. Now run one step of the little learning algorithm and print the new value.

train_step()
print(w)
<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0999993085861206>

It has increased a little bit, from 0 to about 0.1. Now run 1,000 iterations of train_step.

for i in range(1000):
    train_step()
print(w)
<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=5.000000953674316>

It runs pretty quickly, and now \(w\) is nearly 5, which we knew was the minimum of the cost function. Is that not cool? We just specified the cost function and never had to take derivatives; TensorFlow figured out how to minimize it for us. The parameter we wanted to optimize is \(w\), which is why it was declared as a variable, and all we had to do was record the sequence of operations that computes the cost inside a GradientTape; TensorFlow could then automatically work out the derivatives. In TensorFlow you basically only implement the forward prop step, and it figures out the gradient computation.

Getting Training Data into the Program

There is one more feature worth seeing. In the example so far, the cost is a fixed function of the variable \(w\). But when you train a neural network, the function you want to minimize depends not just on the parameters but also on your training data \(x\) (or \(x\) and \(y\)). How do you get training data into a TensorFlow program?

Still define w as a variable and set up the optimizer as before, but now define x as an array of numbers, also float32. The three numbers 1, \(-10\), and 25 will play the role of the coefficients of the cost function, so you can think of x as being like data that controls the coefficients of this quadratic.

w = tf.Variable(0, dtype=tf.float32)
optimizer = tf.keras.optimizers.Adam(0.1)
x = np.array([1.0, -10.0, 25.0], dtype=np.float32)

def training(x, w, optimizer):
    def cost_fn():
        return x[0] * w ** 2 + x[1] * w + x[2]
    for i in range(1000):
        with tf.GradientTape() as tape:
            cost = cost_fn()
        grads = tape.gradient(cost, [w])
        optimizer.apply_gradients(zip(grads, [w]))
    return w

w = training(x, w, optimizer)
print(w)
<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=5.000000953674316>

This computes exactly the same cost function as before, except that the little piece of data in the array x controls the coefficients of the quadratic, and after 1,000 iterations \(w\) again ends up at roughly 5, the minimum. If your cost depended on a real training set, that data would enter the program the same way, as arrays feeding the cost computation.

NoteThe optimizer.minimize Shorthand

In the lecture, one step of training is also written with a simpler alternative piece of syntax that wraps the gradient tape and apply_gradients lines into one call,

optimizer.minimize(cost_fn, [w])

This method existed on optimizers in the TensorFlow version used when the course was filmed. In current TensorFlow releases (which bundle Keras 3), Optimizer.minimize has been removed, and the gradient tape plus apply_gradients pattern shown above is the way to write the same thing, so that is what the executed code here uses.

Computation Graph

What is this code really doing? The heart of the TensorFlow program is something that computes the cost, and TensorFlow automatically figures out the derivatives and how to minimize it. The cost line is what allows TensorFlow to construct a computation graph, which takes \(x_0\) and takes \(w\), squares \(w\), multiplies \(x_0\) with \(w^2\) to give \(x_0 w^2\), and so on through multiple steps until it builds up the computation of the cost function, the last step being to add in the final coefficient \(x_2\).

The nice thing about TensorFlow is that by implementing basically the forward prop through this computation graph, TensorFlow automatically figures out all the necessary backward calculations, all the backward steps needed to implement backprop. That is why you do not need to implement backprop explicitly; TensorFlow figures it out for you.

This is one of the things that makes the programming frameworks so efficient to work with, and a lot can be changed with just one line of code. For example, if you do not want to use the Adam optimizer but a different one, you just change the one line of code that creates the optimizer and swap it out. All the popular modern deep learning frameworks support things like this, and it makes it much easier to develop even pretty complex neural networks.

That is the typical structure of a TensorFlow program. To recap this part of the course, you saw how to systematically organize the hyperparameter search process, you saw batch normalization and how to use it to speed up training, and you learned about deep learning programming frameworks and TensorFlow, which the TensorFlow lab puts into practice.

Review Questions

1. In a TensorFlow program, why do you only need to implement forward prop, and what role does tf.GradientTape play?

Writing the code that computes the cost lets TensorFlow build a computation graph of the operations involved. The gradient tape records the sequence of operations during the forward computation, like a cassette tape, and playing it backwards revisits the operations in reverse order to compute the gradients, so backprop never has to be implemented explicitly.

1. In the example, why is \(w\) declared with tf.Variable, and how does training data enter the program?

tf.Variable signifies the parameters that the optimizer is allowed to change, here the single parameter being optimized, initialized to 0 with type tf.float32. Training data enters as ordinary arrays, like x = np.array([1.0, -10.0, 25.0]), that the cost computation uses; in the example the three numbers control the coefficients of the quadratic \(x_0 w^2 + x_1 w + x_2\), exactly the way a real cost function depends on the training set as well as the parameters.

1. Minimizing \(J(w) = w^2 - 10w + 25\) with Adam at learning rate 0.1 starting from \(w = 0\): what do you observe after one step, and after 1,000 steps?

After one step, \(w\) increases from 0 to about 0.1. After 1,000 iterations, \(w\) is nearly 5, which is the minimum of the cost, since \(J(w) = (w-5)^2\). TensorFlow reached it from the cost specification alone, with no derivatives supplied by hand.

Back to top