TensorFlow笔记


官方文档(中):https://tensorflow.google.cn/programmers_guide/low_level_intro

官方文档(英):https://www.tensorflow.org/guide/low_level_intro


TensorFlow Core Walkthrough

below is from :TensorFlow Core Walkthrough

You might think of TensorFlow Core programs as consisting of two discrete sections:

  1. Building the computational graph (a tf.Graph).
  2. Running the computational graph (using a tf.Session).

A computational graph is a series of TensorFlow operations arranged into a graph. The graph is composed of two types of objects.

  • Operations (or “ops”): The nodes of the graph. Operations describe calculations that consume and produce tensors.
  • Tensors: The edges in the graph. These represent the values that will flow through the graph. Most TensorFlow functions return tf.Tensors.

tf.Tensors do not have values, they are just handles to elements in the computation graph.

These tf.Tensor objects just represent the results of the operations that will be run:

1
2
3
4
5
6
a = tf.constant(3.0, dtype=tf.float32)
b = tf.constant(4.0) # also tf.float32 implicitly
total = a + b
print(a)
print(b)
print(total)

The print statements produce:

1
2
3
Tensor("Const:0", shape=(), dtype=float32)
Tensor("Const_1:0", shape=(), dtype=float32)
Tensor("add:0", shape=(), dtype=float32)

Graph

1
sess.graph.finalize()

网络搭建完成后,将Graph设置为只读,增加或其他修改网络都会报异常,防止内存泄漏。

colocate_gradients_with_ops

有些op的参数是colocate_gradients_with_ops解释

colocate_gradients_with_ops: If True, try colocating gradients with the corresponding op.

tf.data.Dataset

tf.data.Dataset.interleave

关于interleave memory

Main memory divided into two or more sections. The CPU can access alternate sections immediately, without waiting for memory to catch up (through wait states). Interleaved memory is one technique for compensating for the relatively slow speed of dynamic RAM (DRAM). Other techniques include page-mode memory and memory caches.


Re3/training/tf_dataset.py中(PARALLEL_SIZE默认为4):

1
2
dataset = tf.data.Dataset.from_tensor_slices(list(range(PARALLEL_SIZE))).interleave(
get_data_generator, cycle_length=PARALLEL_SIZE)

tf.data.Dataset.interleave belongs to functions that transform an existing dataset, and return a new dataset.

1
2
3
4
5
interleave(
map_func,
cycle_length,
block_length=1
)

Maps map_func across this dataset, and interleaves the results.

Args:

  • map_func: A function mapping a nested structure of tensors (having shapes and types defined by self.output_shapes and self.output_types) to a Dataset.
  • cycle_length: The number of elements from this dataset that will be processed concurrently.
  • block_length: The number of consecutive elements to produce from each input element before cycling to another input element.

Returns:

  • Dataset: A Dataset.

demo:

  1. tf.data.Dataset | TensorFlow

  2. 在Re3中(re3-tensorflow-master/training/tf_dataset.py),tf.data写成类似如下形式:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    PARALLEL_SIZE = 4
    prefetch_size = 64 # batchsize

    def get_data_sequence():
    return np.random.randint(10,100,size=[5, 2]), np.random.randint(0,10,size=[5, 1])

    def generator():
    while True:
    yield get_data_sequence()

    def get_dataset(batch_size):
    def get_data_generator(ind):
    dataset = tf.data.Dataset.from_generator(generator, (tf.uint8, tf.uint8))
    dataset = dataset.prefetch(int(np.ceil(prefetch_size * 1.0 / PARALLEL_SIZE)))
    return dataset

    dataset = tf.data.Dataset.from_tensor_slices(list(range(PARALLEL_SIZE))).interleave(get_data_generator, cycle_length=PARALLEL_SIZE)

    dataset = dataset.batch(batch_size)

    dataset_iterator = dataset.make_one_shot_iterator()

    return dataset_iterator

    刚开始不太理解为什么要这样写:

    dataset = tf.data.Dataset.from_tensor_slices(list(range(PARALLEL_SIZE))).interleave(get_data_generator, cycle_length=PARALLEL_SIZE)

    后来才想明白是生成了PARALLEL_SIZE个generator(),并行生成数据,一起加入dataset中。dataset.prefetch(int(np.ceil(prefetch_size * 1.0 / PARALLEL_SIZE)))正好构成一个prefetch_size的数据.

    做一个实验证明这种方式确实有效,在get_data_sequence()函数中延迟2秒:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    import tensorflow as tf
    import numpy as np
    import time

    PARALLEL_SIZE = 4
    prefetch_size = 8

    def get_data_sequence():
    time.sleep(2)
    return np.random.randint(10, 100, size=[5, 2]), np.random.randint(0, 10, size=[5, 1])

    def generator():
    while True:
    yield get_data_sequence()

    def get_dataset(batch_size):
    def get_data_generator(ind):
    dataset = tf.data.Dataset.from_generator(generator, (tf.uint8, tf.uint8))
    dataset = dataset.prefetch(int(np.ceil(prefetch_size * 1.0 / PARALLEL_SIZE)))
    return dataset

    dataset = tf.data.Dataset.from_tensor_slices(list(range(PARALLEL_SIZE))).interleave(
    get_data_generator, cycle_length=PARALLEL_SIZE)

    # dataset = dataset.batch(batch_size)

    dataset_iterator = dataset.make_one_shot_iterator()

    return dataset_iterator

    if __name__ == '__main__':
    dataset_iterator = get_dataset(batch_size=1)
    imageBatch, labelsBatch = dataset_iterator.get_next()

    with tf.Session() as sess:

    start_time = time.time()

    for _ in range(20):
    iter_start = time.time()
    sess.run([imageBatch, labelsBatch])
    print('loop time spend:', time.time() - iter_start, 's')

    print('total time spend:', time.time()-start_time)

    输出:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    loop time spend: 2.0119364261627197 s
    loop time spend: 2.0057930946350098 s
    loop time spend: 2.004932403564453 s
    loop time spend: 2.0054099559783936 s
    loop time spend: 0.0005707740783691406 s
    loop time spend: 0.0004899501800537109 s
    loop time spend: 0.0003542900085449219 s
    loop time spend: 2.0011227130889893 s
    loop time spend: 0.0006690025329589844 s
    loop time spend: 0.00034117698669433594 s
    loop time spend: 0.00029540061950683594 s
    loop time spend: 2.0007288455963135 s
    loop time spend: 0.0005996227264404297 s
    loop time spend: 0.0003237724304199219 s
    loop time spend: 0.00043082237243652344 s
    loop time spend: 2.0010249614715576 s
    loop time spend: 0.0004963874816894531 s
    loop time spend: 0.0002396106719970703 s
    loop time spend: 0.00018930435180664062 s
    loop time spend: 2.00167179107666 s

    total time spend: 16.03883385658264

    PARALLEL_SIZE=4,有4个核心在同时生成数据可以看出,前四组数据不能prefetch,所以都耗时2s,当第四个核心生成数据后,前三个的已经准备好了,所以后面三组数据耗时为0.

    个人理解:

    • PARALLEL_SIZE的最大值最好设置为CPU核心数

tf.data.Dataset.repeat

tf.data.Dataset.repeat

demo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import numpy as np
import tensorflow as tf

dataset = tf.data.Dataset.from_tensor_slices(
{
"a": np.array([1.0, 2.0, 3.0, 4.0, 5.0]),
"b": np.random.uniform(size=(5, 2))
})

dataset = dataset.repeat(2) # <-----

iterator = dataset.make_one_shot_iterator()
one_element = iterator.get_next()
with tf.Session() as sess:
try:
while True:
print(sess.run(one_element))
except tf.errors.OutOfRangeError:
print("end!")

输出:

1
2
3
4
5
6
7
8
9
10
11
{'a': 1.0, 'b': array([0.0757098 , 0.26210605])}
{'a': 2.0, 'b': array([0.9751478 , 0.92521069])}
{'a': 3.0, 'b': array([0.38879417, 0.3015516 ])}
{'a': 4.0, 'b': array([0.95576306, 0.98980403])}
{'a': 5.0, 'b': array([0.18889013, 0.77938525])}
{'a': 1.0, 'b': array([0.0757098 , 0.26210605])}
{'a': 2.0, 'b': array([0.9751478 , 0.92521069])}
{'a': 3.0, 'b': array([0.38879417, 0.3015516 ])}
{'a': 4.0, 'b': array([0.95576306, 0.98980403])}
{'a': 5.0, 'b': array([0.18889013, 0.77938525])}
end!

repeat(2)改为repeat(3),输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{'a': 1.0, 'b': array([0.46560799, 0.94307949])}
{'a': 2.0, 'b': array([0.99846131, 0.03052336])}
{'a': 3.0, 'b': array([0.85013789, 0.71297778])}
{'a': 4.0, 'b': array([0.87352444, 0.58657701])}
{'a': 5.0, 'b': array([0.31829909, 0.1906395 ])}
{'a': 1.0, 'b': array([0.46560799, 0.94307949])}
{'a': 2.0, 'b': array([0.99846131, 0.03052336])}
{'a': 3.0, 'b': array([0.85013789, 0.71297778])}
{'a': 4.0, 'b': array([0.87352444, 0.58657701])}
{'a': 5.0, 'b': array([0.31829909, 0.1906395 ])}
{'a': 1.0, 'b': array([0.46560799, 0.94307949])}
{'a': 2.0, 'b': array([0.99846131, 0.03052336])}
{'a': 3.0, 'b': array([0.85013789, 0.71297778])}
{'a': 4.0, 'b': array([0.87352444, 0.58657701])}
{'a': 5.0, 'b': array([0.31829909, 0.1906395 ])}
end!

注意,tf.data.Dataset.make_one_shot_iterator creates an Iterator for enumerating the elements of this dataset. 不是将np.array([1.0, 2.0, 3.0, 4.0, 5.0])进行重复操作!对于"b": np.random.uniform(size=(5, 2)),将其最外层的元素进行重复操作。

repeat的功能就是将整个序列重复多次,主要用来处理机器学习中的epoch,假设原先的数据是一个epoch,使用repeat(2)就可以将之变成2个epoch.

如果直接调用repeat( )的话,生成的序列就会无限重复下去,没有结束,因此也不会抛出tf.errors.OutOfRangeError异常。

tf.data.Dataset.batch

tf.data.Dataset.batch

demo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import numpy as np
import tensorflow as tf

dataset = tf.data.Dataset.from_tensor_slices(
{
"a": np.array([1.0, 2.0, 3.0, 4.0, 5.0]),
"b": np.random.uniform(size=(5, 2))
})

dataset = dataset.batch(2) # <-----

iterator = dataset.make_one_shot_iterator()
one_element = iterator.get_next()
with tf.Session() as sess:
try:
while True:
print(sess.run(one_element))
except tf.errors.OutOfRangeError:
print("end!")

输出:

1
2
3
4
5
6
{'a': array([1., 2.]), 'b': array([[0.76340145, 0.9742068 ],
[0.65718591, 0.88166876]])}
{'a': array([3., 4.]), 'b': array([[0.39534227, 0.46811893],
[0.00185565, 0.70567686]])}
{'a': array([5.]), 'b': array([[0.82289004, 0.48191698]])}
end!

batch(2)改为batch(3),输出为:

1
2
3
4
5
6
{'a': array([1., 2., 3.]), 'b': array([[0.22007263, 0.14702525],
[0.35594734, 0.0250401 ],
[0.02699881, 0.0258704 ]])}
{'a': array([4., 5.]), 'b': array([[0.13059928, 0.25843699],
[0.45355536, 0.07123547]])}
end!

make_one_shot_iterator() creates an Iterator for enumerating the elements of this dataset. 是内部的元素,不是np.array([1.0, 2.0, 3.0, 4.0, 5.0])整体。

注意,当超过迭代器范围时,出现tf.errors.OutOfRangeError异常。

tf.data.Dataset.from_generator


Re3/training/tf_dataset.py中:

1
dataset = tf.data.Dataset.from_generator(self.generator, (tf.uint8, tf.float32))

tf.data.Dataset.from_generator creates a Dataset whose elements are generated by generator.

demo:

from_generator

Tensorflow Dataset.from_generator使用示例:

我们知道,tensorflow的基本原理是先构造一个计算图,最后再统一计算。为此,tf重写了几乎所有常见函数,用于构造计算图,而且tensorflow不支持循环、选择等普通编程语言的常见操作。这就给编程使用带来比较大的麻烦。具体到data feeding上,也是如此。虽然设计了placeholder、train.slice_input_producer系列、Dataset等多种方式,但使用中仍有各种不便,尤其是在输入形式复杂、需要多重变换的时候更是如此。而Dataset.from_generator可以在一定程度上解决这个问题。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import numpy as np
import tensorflow as tf

def data_generator():
dataset = np.array(range(5))
for d in dataset:
yield d

dataset = tf.data.Dataset.from_generator(data_generator, (tf.int32))
# dataset = dataset.repeat(3)
# dataset = dataset.batch(4)

iterator = dataset.make_one_shot_iterator()
one_element = iterator.get_next()

with tf.Session() as sess:
try:
batch_num=0
while True:
one_batch = sess.run(one_element)
print('Batch No. %d:' % batch_num)
print(one_batch)
print('')
batch_num+=1

except tf.errors.OutOfRangeError:
print('end!')

输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Batch No. 0:
0

Batch No. 1:
1

Batch No. 2:
2

Batch No. 3:
3

Batch No. 4:
4

end!

单独取消dataset = dataset.repeat(3)的注释,输出结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Batch No. 0:
0

Batch No. 1:
1

Batch No. 2:
2

Batch No. 3:
3

Batch No. 4:
4

Batch No. 5:
0

Batch No. 6:
1

Batch No. 7:
2

Batch No. 8:
3

Batch No. 9:
4

Batch No. 10:
0

Batch No. 11:
1

Batch No. 12:
2

Batch No. 13:
3

Batch No. 14:
4

end!

单独取消dataset = dataset.batch(4)的注释,输出结果:

1
2
3
4
5
6
7
Batch No. 0:
[0 1 2 3]

Batch No. 1:
[4]

end!

同时取消二者注释,输出结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
Batch No. 0:
[0 1 2 3]

Batch No. 1:
[4 0 1 2]

Batch No. 2:
[3 4 0 1]

Batch No. 3:
[2 3 4]

end!

注意,repeatbatch操作都是在dataset.make_one_shot_iterator()之前的。

tf.data.Dataset.prefetch

参考:

  1. tf.data.Dataset.prefetch

  2. Input Pipeline Performance Guide

  3. TensorFlow 如何构建高性能的数据输入管道(Pipeline)(中文翻译)

    tf.data API 通过 tf.data.Dataset.prefetch 变换提供了一个 software pipelining 机制,这个机制解耦了 数据产生的时间 和 数据消耗的时间。尤其是,这个机制使用一个后台线程和一个内部缓存区,在数据被请求前,去从数据数据集中预加载一些数据。

参考资料:

  1. 导入数据 | TensorFlow(中)

    Importing Data(英)

  2. Dataset Input Pipeline | TensorFlow

  3. 『TensorFlow』数据读取类_data.Dataset

Tensor name

How does TensorFlow name tensors?

关于tensor name后面的:0,上面的链接可以解释,实际上表示op的第几个输出。举例:

1
print(tf.split([0, 1, 2, 3, 4, 5], 6, name="split_op")[3].name)

输出:

1
split_op:3

name_scope和variable_scope

参考:https://www.cnblogs.com/MY0213/p/9208503.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
with tf.name_scope("my_scope"):
v1 = tf.get_variable("var1", [1], dtype=tf.float32)
v2 = tf.Variable(1, name="var2", dtype=tf.float32)

with tf.variable_scope("my_scope"):
v3 = tf.get_variable("var1", [1], dtype=tf.float32)
v4 = tf.Variable(1, name="var2", dtype=tf.float32)

print("v1:",v1)
print("v2:",v2)
print("v3:",v3)
print("v4:",v4)

--------

v1: <tf.Variable 'var1:0' shape=(1,) dtype=float32_ref>
v2: <tf.Variable 'my_scope/var2:0' shape=() dtype=float32_ref>
v3: <tf.Variable 'my_scope/var1:0' shape=(1,) dtype=float32_ref>
v4: <tf.Variable 'my_scope_1/var2:0' shape=() dtype=float32_ref>

name_scope对tf.get_variable不起作用,对tf.Variable起作用。

tf.Variable会自动检测命名冲突并自行处理。tf.get_variable遇到重名的变量创建且变量名没有设置为共享变量时,则会报错。

可以使用name_scope给operation分类。使用variable_scope来区分variable.

var1:0中冒号后的0表示这个operation的第几个输出,从0开始,0,1,2,…

tf.name_scope(None) 有清除name scope的作用:

1
2
3
4
5
6
7
8
9
10
import tensorflow as tf
with tf.name_scope("hehe"):
w1 = tf.Variable(1.0)
with tf.name_scope(None):
w2 = tf.Variable(2.0)
print(w1.name)
print(w2.name)

#hehe/Variable:0
#Variable:0

reuse

1
2
3
4
5
6
7
8
9
10
import tensorflow as tf

def create_variable(reuse=False):
with tf.variable_scope("foo", reuse=reuse):
with tf.variable_scope("bar"):
v = tf.get_variable("v", [1])
return v


a = create_variable(reuse=True)

会提示错误:

1
ValueError: Variable foo/bar/v does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?

报错是因为,tf.variable_scope("foo", reuse=reuse)使用了reuse参数,必须已经存在了scope范围内的这些变量,也就是说第一次创建变量不能用reuse=reuse,但是可以用reuse=reuse=tf.AUTO_REUSE(相当于第一次创建变量时正常创建,后面都用reuse)。

How to set variable reuse back to False in Tensorflow?

1
2
3
4
5
print tf.get_variable_scope().reuse
with tf.variable_scope(tf.get_variable_scope(), reuse=True):
print tf.get_variable_scope().reuse
# Code that reuse variables goes here
print tf.get_variable_scope().reuse

输出:

1
2
3
False
True
False

tf.get_variable_scope()能获取这行语句当前的scope,通过这种方法可以在with语句里面对不可重用的变量进行重用。

get_collection

tf.get_collection

1
2
3
4
tf.get_collection(
key,
scope=None
)

注意scope参数:

  • scope: (Optional.) If supplied, the resulting list is filtered to include only items whose name attribute matches using re.match. Items without a name attribute are never returned if a scope is supplied and the choice or re.match means that a scope without special tokens filters by prefix.

tf.assign

https://tensorflow.google.cn/api_docs/python/tf/assign

1
2
3
4
5
6
7
tf.assign(
ref,
value,
validate_shape=None,
use_locking=None,
name=None
)

将value赋值给ref

This operation outputs a Tensor that holds the new value of ‘ref’ after the value has been assigned. This makes it easier to chain operations that need to use the reset value.

保存和恢复

官方文档:https://tensorflow.google.cn/programmers_guide/saved_model

checkpoints

保存和恢复变量

从已保存的变量中选择恢复(比如选择一些层的变量恢复)

已经保存的文件格式如:

1
2
3
4
checkpoint             
model.ckpt-260946.data-00000-of-00001
model.ckpt-260946.index
model.ckpt-260946.meta

260946表示迭代次数(在保存时可选将其加入文件名),保存一次迭代额的数据就会产生对应上面的三个文件,checkpoint中是几次保存的信息。

一个简单的权重恢复例子(注意这里定义的权重变量名和保存的要相同):

1
2
3
4
5
6
7
8
9

build_net_work()

restored_variables = tf.trainable_variables()

saver = tf.train.Saver(restored_variables)

with tf.Session() as sess:
saver.restore(sess, "./checkpoints/model.ckpt-260946")

通过tf.trainable_variables()获取可训练的tensor变量,传入saver类的对象创建中,在执行restore方法时,注意第二个参数不是实际文件名(去掉实际文件命的最后一个后缀)。

get_checkpoint_state

如果checkpoint文件中的内容为:

1
2
3
model_checkpoint_path: "/home/ubuntu/DeepLearningModel/QA/logs/checkpoints/model.ckpt-381500"
all_model_checkpoint_paths: "/home/ubuntu/DeepLearningModel/QA/logs/checkpoints/model.ckpt-381400"
all_model_checkpoint_paths: "/home/ubuntu/DeepLearningModel/QA/logs/checkpoints/model.ckpt-381500"

表明当前文件下的文件是:

1
2
3
4
5
6
7
checkpoint
model.ckpt-381400.data-00000-of-00001
model.ckpt-381400.index
model.ckpt-381400.meta
model.ckpt-381500.data-00000-of-00001
model.ckpt-381500.index
model.ckpt-381500.meta

checkpoint文件中的语句表示可控恢复的有第381400次迭代和381500次迭代的权重 (最新的是381500次) 。

tf.train.get_checkpoint_state可以获取checkpoint文件中的内容,通过.model_checkpoint_path.all_model_checkpoint_paths分别访问文件中的对应内容:

1
2
3
4
5
ckpt = tf.train.get_checkpoint_state(folder_path)
if ckpt and ckpt.model_checkpoint_path:
pass
for path in ckpt.all_model_checkpoint_paths:
pass

if ckpt是因为get_checkpoint_state读取错误时返回None,算一个出错判断。

NewCheckpointReader

tf.train.NewCheckpointReader(‘path’),官方文档并没有详细的说明,谷歌一下,这个方法可以提取checkpoint中变量的名字和值

1
2
3
4
5
6
7
8
9
import tensorflow as tf

checkpoint_path = os.path.join(model_dir, "model.ckpt")
reader = tf.train.NewCheckpointReader(checkpoint_path)
var_to_shape_map = reader.get_variable_to_shape_map()

for key in var_to_shape_map:
print("tensor_name: ", key)
print(reader.get_tensor(key)) # Remove this is you want to print only variable names

这种方法不需要run!

ndarray和tensor

Any tensor returned by Session.run or eval is a NumPy array.

sess.run

1
a = sess.run(x, feed_dict=feed_dict)

和:

1
b = sess.run([x],feed_dict=feed_dict)

b[0]a相等。

数组维度问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import numpy as np
import tensorflow as tf

# shepe: (3, 4, 2)
a = np.array([[[1.0, 2.0], [1.0, 2.0], [1.0, 2.0], [1.0, 2.0]],
[[1.0, 2.0], [1.0, 2.0], [1.0, 2.0], [1.0, 2.0]],
[[1.0, 2.0], [1.0, 2.0], [1.0, 2.0], [1.0, 2.0]]])

b = tf.constant(a)

mean = tf.reduce_mean(b, axis=(0, 1))

normalize = tf.nn.l2_normalize(b, axis=(0,1))

with tf.Session() as sess:
mean_output, normalize_output = sess.run([mean, normalize])

print(mean_output)
print(normalize_output)

输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[1. 2.]

[[[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]]

[[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]]

[[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]
[0.28867513 0.28867513]]]

注意,tf.reduce_meantf.nn.l2_normalize的参数axis=(0, 1)。分别按每个通道求均值和规范化(12个1和2的规范化都是0.28867513)


----------over----------