Computational graph¶
-
class
blocks.graph.
ComputationGraph
(outputs)[source]¶ Bases:
object
Encapsulates a managed Theano computation graph.
This implies that it not only contains the variables required to compute the given outputs, but also all the auxiliary variables and updates that were attached to these variables through the annotation system.
All variables are presented in topologically sorted order according to the apply nodes that they are an input to.
Parameters: outputs ((list of) TensorVariable
) – The output(s) of the computation graph.-
inputs
¶ list of
TensorVariable
– The inputs of the computation graph. This does not include shared variables and constants.
list of
TensorSharedVariable
– All the shared variables in the graph.
-
parameters
¶ list of
TensorSharedVariable
– All the shared variables which have thePARAMETER
role.
-
outputs
¶ list of
TensorVariable
– The outputs of the computations graph (as passed to the constructor).
-
auxiliary_variables
¶ list of
TensorVariable
– All variables which have theAUXILIARY
role.
-
intermediary_variables
¶ list of
TensorVariable
– Any variable that is not part ofinputs
oroutputs
.
-
variables
¶ list of
TensorVariable
– All variables (including auxiliary) in the managed graph.
-
scans
¶ list of
Scan
– All Scan ops used in this computation graph.
-
scan_variables
¶ list of
TensorVariable
– All variables of the inner graphs of Scan ops.
-
updates
¶ TensorSharedVariable
updates – All the updates found attached to the annotations.
-
auxiliary_variables
-
get_snapshot
(data)[source]¶ Evaluate all role-carrying Theano variables on given data.
Parameters: data (dict of (data source, data) pairs) – Data for input variables. The sources should match with the names of the input variables. Returns: Return type: Dictionary of (variable, variable value on given data) pairs.
-
get_theano_function
(additional_updates=None, **kwargs)[source]¶ Create Theano function from the graph contained.
Parameters: **kwargs (dict) – Keyword arguments to theano.function. Useful for specifying compilation modes or profiling.
-
has_inputs
(variable)[source]¶ Check if a variable depends on input variables.
Returns: True
if the given variable depends on input variables,False
otherwise.Return type: bool
-
inputs
Inputs to the graph, excluding constants and shared variables.
-
intermediary_variables
-
parameters
-
replace
(replacements)[source]¶ Replace certain variables in the computation graph.
Parameters: replacements (dict) – The mapping from variables to be replaced to the corresponding substitutes. Examples
>>> import theano >>> from theano import tensor, function >>> x = tensor.scalar('x') >>> y = x + 2 >>> z = y + 3 >>> a = z + 5
Let’s suppose we have dependent replacements like
>>> replacements = {y: x * 2, z: y * 3} >>> cg = ComputationGraph([a]) >>> theano.pprint(a) '(((x + TensorConstant{2}) + TensorConstant{3}) + TensorConstant{5})' >>> cg_new = cg.replace(replacements) >>> theano.pprint( ... cg_new.outputs[0]) '(((x * TensorConstant{2}) * TensorConstant{3}) + TensorConstant{5})'
First two sums turned into multiplications
>>> float(function(cg_new.inputs, cg_new.outputs)(3.)[0]) 23.0
-
scan_variables
Variables of Scan ops.
-
shared_variables
-
-
blocks.graph.
apply_dropout
(computation_graph, variables, drop_prob, rng=None, seed=None, custom_divisor=None)[source]¶ Apply dropout to specified variables in a graph.
Parameters: - computation_graph (instance of
ComputationGraph
) – The computation graph. - variables (list of
TensorVariable
) – Variables to be dropped out. - drop_prob (float) – Probability of dropping out. If you want to apply the dropout with different probabilities for different layers, call it several times.
- rng (
MRG_RandomStreams
) – Random number generator. - seed (int) – Random seed to be used if rng was not specified.
- custom_divisor (float or None, optional) – Divide dropped variables by a given scalar value. If None, (default) dropped variables will be divided by (1 - drop_prob) which is equivalent to scaling by (1 - drop_prob) at test time as recommended in [DROPOUT].
Returns: dropped_computation_graph – A new computation graph with dropout applied to the specified variables. In order to train with, or monitor, the outputs of the original computation graph with dropout applies, use the variables contained in dropped_computation_graph.outputs.
Return type: instance of
ComputationGraph
Notes
For more information, see [DROPOUT].
[DROPOUT] (1, 2) Hinton et al. Improving neural networks by preventing co-adaptation of feature detectors, arXiv:1207.0580. Examples
>>> import numpy >>> from theano import tensor, function >>> from blocks.bricks import MLP, Identity >>> from blocks.filter import VariableFilter >>> from blocks.initialization import Constant >>> from blocks.roles import INPUT >>> linear = MLP([Identity(), Identity()], [2, 10, 2], ... weights_init=Constant(1), biases_init=Constant(2)) >>> x = tensor.matrix('x') >>> y = linear.apply(x) >>> cg = ComputationGraph(y)
We are going to drop out all the input variables
>>> inputs = VariableFilter(roles=[INPUT])(cg.variables)
Here we apply dropout with default setting to our computation graph
>>> cg_dropout = apply_dropout(cg, inputs, 0.5)
Dropped out variables have role DROPOUT and are tagged with replacement_of tag. Let’s filter these variables and check if they have the links to original ones.
>>> dropped_out = VariableFilter(roles=[DROPOUT])(cg_dropout.variables) >>> inputs_referenced = [var.tag.replacement_of for var in dropped_out] >>> set(inputs) == set(inputs_referenced) True
Compiling theano functions to forward propagate in original and dropped out graphs
>>> fprop = function(cg.inputs, cg.outputs[0]) >>> fprop_dropout = function(cg_dropout.inputs, cg_dropout.outputs[0])
Initialize an MLP and apply these functions
>>> linear.initialize() >>> fprop(numpy.ones((3, 2), ... dtype=theano.config.floatX)) array([[ 42., 42.], [ 42., 42.], [ 42., 42.]]... >>> fprop_dropout(numpy.ones((3, 2), ... dtype=theano.config.floatX)) array([[ 0., 0.], [ 0., 0.], [ 0., 0.]]...
And after the second run answer is different
>>> fprop_dropout(numpy.ones((3, 2), ... dtype=theano.config.floatX)) array([[ 0., 52.], [ 100., 0.], [ 0., 0.]]...
- computation_graph (instance of
-
blocks.graph.
apply_noise
(computation_graph, variables, level, seed=None)[source]¶ Add Gaussian noise to certain variable of a computation graph.
Parameters: - computation_graph (instance of
ComputationGraph
) – The computation graph. - variables (
TensorVariable
) – Variables to add noise to. - level (float) – Noise level.
- seed (int, optional) – The seed with which
MRG_RandomStreams
is initialized, is set to 1 by default.
- computation_graph (instance of
-
blocks.graph.
collect_parameters
(computation_graph, parameters)[source]¶ Replace parameters with a single shared variable.
This can be useful if you need to calculate the full Hessian of a computational graph. It replaces parameters with slices of a single large vectors like
>>> from blocks.utils import shared_floatx >>> W1 = shared_floatx(numpy.random.rand(10, 10)) >>> W2 = shared_floatx(numpy.random.rand(10, 10)) >>> all_parameters = shared_floatx(numpy.concatenate( ... [W1.get_value().flatten(), W2.get_value().flatten()])) >>> W1 = all_parameters[:W1.size] >>> W2 = all_parameters[W1.size:]
Parameters: - computation_graph (
ComputationGraph
instance) – The managed Theano graph in which to collect parameters. - parameters (list of Theano shared variables) – The parameters whose values should be collected.
Returns: A new Theano graph which has all the given parameters collected into a single large shared variable.
Return type: ComputationGraph instance
Notes
Note that this replacement makes the training of the model significantly slower because of the large amount of Theano’s
set_subtensor
calls needed to train the model.Examples
>>> from blocks.bricks import MLP, Logistic >>> from blocks.bricks.cost import SquaredError >>> from theano import tensor >>> x = tensor.matrix() >>> mlp = MLP(activations=[Logistic(), Logistic()], ... dims=[784, 100, 784]) >>> cost = SquaredError().apply(x, mlp.apply(x)) >>> cg = ComputationGraph(cost) >>> new_cg = collect_parameters(cg, cg.shared_variables)
The new graph only has a single shared variable. This variable receives the
COLLECTOR
role.>>> new_cg.shared_variables [collected_parameters]
The bricks’ variables have been replaced with reshaped segments of this single shared variable. These replacements are given the
COLLECTED
role.>>> from blocks.filter import VariableFilter >>> from blocks.roles import PARAMETER >>> var_filter = VariableFilter(roles=[COLLECTED]) >>> var_filter(new_cg.variables) [Reshape{1}.0, Reshape{1}.0, Reshape{2}.0, Reshape{2}.0]
- computation_graph (