Computation Graph"

Overview

To use TensorFlow you need to understand how TensorFlow:

• Represents computations as graphs.
• Executes graphs in the context of Sessions.
• Represents data as tensors.
• Maintains state with Variables.
• Uses feeds and fetches to get data into and out of arbitrary operations.

TensorFlow is a programming system in which you represent computations as graphs. Nodes in the graph are called ops (short for operations). An op takes zero or more Tensors, performs some computation, and produces zero or more Tensors. A Tensor is a typed multi-dimensional array. For example, you can represent a mini-batch of images as a 4-D array of floating point numbers with dimensions shape(batch, height, width, channels).

A TensorFlow graph is a description of computations. To compute anything, a graph must be launched in a Session. A Session places the graph ops onto Devices, such as CPUs or GPUs, and provides methods to execute them. These methods return tensors produced by ops as R vectors, matrices, and multi-dimensional arrays.

This article includes further explanation and short examples for each of these concepts. The article on Using the TensorFlow API from R provides additional details on the R API, including how to access help and additional documentation.

Computation Graph

TensorFlow programs are usually structured into a construction phase, that assembles a graph, and an execution phase that uses a session to execute ops in the graph.

For example, it is common to create a graph to represent and train a neural network in the construction phase, and then repeatedly execute a set of training ops in the graph in the execution phase.

library(tensorflow)

# Create a Constant op that produces a 1x2 matrix.  The op is
# added as a node to the default graph.
#
# The value returned by the constructor represents the output
# of the Constant op.
matrix1 <- tf$constant(matrix(c(3.0, 3.0), nrow = 1, ncol = 2))

# Create another Constant that produces a 2x1 matrix.
matrix2 <- tf$constant(matrix(c(3.0, 3.0), nrow = 2, ncol = 1))

# Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.
# The returned value, 'product', represents the result of the matrix
# multiplication.
product <- tf$matmul(matrix1, matrix2)

# Launch the default graph.
sess <- tf$Session()

# To run the matmul op we call the session 'run()' method, passing 'product'
# which represents the output of the matmul op.  This indicates to the call
# that we want to get the output of the matmul op back.
#
# All inputs needed by the op are run automatically by the session.  They
# typically are run in parallel.
#
# The call 'run(product)' thus causes the execution of three ops in the
# graph: the two constants and matmul.
#
# The output of the op is returned in 'result' as a 1x1 matrix.
result <- sess$run(product)
print(result)

# Close the Session when we're done.
sess$close()

with(tf$Session() %as% sess, {
  result = sess$run(product)
  print(result)
})

with(tf$Session() %as% sess, {
  with(tf$device("/gpu:1"), {
    matrix1 <- tf$constant(matrix(c(3.0, 3.0), nrow = 1, ncol = 2))
    matrix2 <- tf$constant(matrix(c(3.0, 3.0), nrow = 2, ncol = 1))
    product <- tf$matmul(matrix1, matrix2)
  })
})

with(tf$Session("grpc://example.org:2222") %as% sess, {
   # Calls to sess.run(...) will be executed on the cluster.
})

with(tf$device("/job:ps/task:0"), {
  weights <- tf$Variable(...)
  biases <- tf$Variable(...)
})

Interactive Usage

The examples in the documentation launch the graph with a Session and use the Session$run() method to execute operations.

For ease of use in interactive R shells, you can instead use the InteractiveSession class, and the Tensor$eval() and Operation$run() methods. This avoids having to keep a variable holding the session.

# Enter an interactive TensorFlow Session.
library(tensorflow)
sess <- tf$InteractiveSession()

x <- tf$Variable(c(1.0, 2.0))
a <- tf$constant(c(3.0, 3.0))

# Initialize 'x' using the run() method of its initializer op.
x$initializer$run()

# Add an op to subtract 'a' from 'x'.  Run it and print the result
sub <- tf$subtract(x, a)
print(sub$eval())

# Close the Session when we're done.
sess$close()

Tensors

TensorFlow programs use a tensor data structure to represent all data – only tensors are passed between operations in the computation graph. You can think of a TensorFlow tensor as an n-dimensional array or list. A tensor has a static type, a rank, and a shape. To learn more about how TensorFlow handles these concepts, see the Rank, Shape, and Type reference.

Variables

Variables maintain state across executions of the graph. The following example shows a variable serving as a simple counter. See Variables for more details.

# Create a Variable, that will be initialized to the scalar value 0.
state <- tf$Variable(0L, name="counter")

# Create an Op to add one to `state`.
one <- tf$constant(1L)
new_value <- tf$add(state, one)
update <- tf$assign(state, new_value)

# Variables must be initialized by running an `init` Op after having
# launched the graph.  We first have to add the `init` Op to the graph.
init_op <- tf$global_variables_initializer()

# Launch the graph and run the ops.
with(tf$Session() %as% sess, {
   # Run the 'init' op
  sess$run(init_op)
  # Print the initial value of 'state'
  print(sess$run(state))
  # Run the op that updates 'state' and print 'state'.
  for (i in 1:3) {
    sess$run(update)
    print(sess$run(state))
  }
})

The assign() operation in this code is a part of the expression graph just like the add() operation, so it does not actually perform the assignment until run() executes the expression.

You typically represent the parameters of a statistical model as a set of Variables. For example, you would store the weights for a neural network as a tensor in a Variable. During training you update this tensor by running a training graph repeatedly.

Fetches

To fetch the outputs of operations, execute the graph with a run() call on the Session object and pass in the tensors to retrieve. In the previous example we fetched the single node state, but you can also fetch multiple tensors:

input1 <- tf$constant(3.0)
input2 <- tf$constant(2.0)
input3 <- tf$constant(5.0)
intermed <- tf$add(input2, input3)
mul <- tf$multiply(input1, intermed)

with(tf$Session() %as% sess, {
  result = sess$run(list(mul, intermed))
  print(result)
})

All the ops needed to produce the values of the requested tensors are run once (not once per requested tensor).

Feeds

The examples above introduce tensors into the computation graph by storing them in Constants and Variables. TensorFlow also provides a feed mechanism for patching a tensor directly into any operation in the graph.

A feed temporarily replaces the output of an operation with a tensor value. You supply feed data as an argument to a run() call. The feed is only used for the run call to which it is passed. The most common use case involves designating specific operations to be “feed” operations by using tf$placeholder() to create them:

input1 <- tf$placeholder(tf$float32)
input2 <- tf$placeholder(tf$float32)
output <- tf$multiply(input1, input2)

with(tf$Session() %as% sess, {
  print(sess$run(output, feed_dict=dict(input1 = 7.0, input2 = 2.0)))
})

A placeholder() operation generates an error if you do not supply a feed for it. See the MNIST fully-connected feed tutorial (source code) for a larger-scale example of feeds.