详解DNN训练中出现的问题与解决方法方法
由于深度神经网络（dnn）层数很多，每次训练都是逐层由后至前传递。传递项1梯度非常大，以此修正网络会不断震荡，无法形成一个收敛网络。因而dnn的训练中可以形成很多tricks。。
1、初始化权重 起初采用正态分布随机化初始权重，会使得原本单位的variance逐渐变得非常大。例如下图的sigmoid函数，靠近0点的梯度近似线性很敏感，但到了，即很强烈的输入产生木讷的输出。
采用xavier initialization，根据fan-in（输入神经元个数）和fan-out（输出神经元个数）设置权重。
并设计针对不同激活函数的初始化策略，如下图（左边是均态分布，右边正态分布较为常用）
2、激活函数 一般使用relu，但是不能有小于0的输入（dying relus）
a.leaky relu
改进方法leaky relu=max(αx,x)，小于0时保留一点微小特征。
具体应用
from tensorflow.examples.tutorials.mnist import input_datamnist = input_data.read_data_sets(/tmp/data/)reset_graph()n_inputs =28*28# mnistn_hidden1 =300n_hidden2 =100n_outputs =10x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=leaky_relu, name=hidden1) hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=leaky_relu, name=hidden2) logits =tf.layers.dense(hidden2, n_outputs, name=outputs)withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)learning_rate =0.01withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32))init =tf.global_variables_initializer()saver =tf.train.saver()n_epochs =40batch_size =50withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) ifepoch %5==0: acc_train = accuracy.eval(feed_dict={x: x_batch,y: y_batch}) acc_test = accuracy.eval(feed_dict={x: mnist.validation.images,y: mnist.validation.labels}) print(epoch,batch accuracy:, acc_train,validation accuracy:, acc_test) save_path = saver.save(sess,./my_model_final.ckpt) b. elu改进
另一种改进elu，在神经元小于0时采用指数变化
#just specify the activation function when building each layerx= tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)hidden1= tf.layers.dense(x, n_hidden1, activation=tf.nn.elu, name=hidden1) c. selu
最新提出的是selu（仅给出关键代码）
withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=selu, name=hidden1) hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=selu, name=hidden2) logits =tf.layers.dense(hidden2, n_outputs, name=outputs)# train 过程means = mnist.train.images.mean(axis=0, keepdims=true)stds = mnist.train.images.std(axis=0, keepdims=true) +1e-10withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) x_batch_scaled = (x_batch - means) / stds sess.run(training_op, feed_dict={x: x_batch_scaled,y: y_batch}) ifepoch %5==0: acc_train = accuracy.eval(feed_dict={x: x_batch_scaled,y: y_batch}) x_val_scaled = (mnist.validation.images - means) / stds acc_test = accuracy.eval(feed_dict={x: x_val_scaled,y: mnist.validation.labels}) print(epoch,batch accuracy:, acc_train,validation accuracy:, acc_test) save_path = saver.save(sess,./my_model_final_selu.ckpt)3、batch normalization 在2015年，有研究者提出，既然使用mini-batch进行操作，对每一批数据也可采用，在调用激活函数之前，先做一下normalization，使得输出数据有一个较好的形状，初始时，超参数scaling（γ）和shifting（β）进行适度缩放平移后传递给activation函数。步骤如下：
现今batch normalization已经被tensorflow实现成一个单独的层，直接调用
测试时，由于没有mini-batch，故训练时直接使用训练时的mean和standard deviation（），实现代码如下
import tensorflowastfn_inputs =28*28n_hidden1 =300n_hidden2 =100n_outputs =10batch_norm_momentum =0.9x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)training =tf.placeholder_with_default(false, shape=(), name='training')withtf.name_scope(dnn): he_init =tf.contrib.layers.variance_scaling_initializer() #相当于单独一层 my_batch_norm_layer = partial( tf.layers.batch_normalization, training=training, momentum=batch_norm_momentum) my_dense_layer = partial( tf.layers.dense, kernel_initializer=he_init) hidden1 = my_dense_layer(x, n_hidden1, name=hidden1) bn1 =tf.nn.elu(my_batch_norm_layer(hidden1))# 激活函数使用elu hidden2 = my_dense_layer(bn1, n_hidden2, name=hidden2) bn2 =tf.nn.elu(my_batch_norm_layer(hidden2)) logits_before_bn = my_dense_layer(bn2, n_outputs, name=outputs) logits = my_batch_norm_layer(logits_before_bn)# 输出层也做一个batch normalizationwithtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32)) init =tf.global_variables_initializer()saver =tf.train.saver()n_epochs =20batch_size =200#需要显示调用训练时得出的方差均值，需要额外调用这些算子extra_update_ops =tf.get_collection(tf.graphkeys.update_ops)#在training和testing时不一样withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run([training_op, extra_update_ops], feed_dict={training:true,x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_model_final.ckpt)4、gradient clipp 处理gradient之后往后传，一定程度上解决梯度爆炸问题。（但由于有了batch normalization，此方法用的不多）
threshold =1.0optimizer = tf.train.gradientdescentoptimizer(learning_rate)grads_and_vars = optimizer.compute_gradients(loss)capped_gvs = [(tf.clip_by_value(grad, -threshold, threshold),var) forgrad,varingrads_and_vars]training_op = optimizer.apply_gradients(capped_gvs)5、重用之前训练过的层 （reusing pretrained layers）
对之前训练的模型稍加修改，节省时间，在深度模型训练（由于有很多层）中经常使用。
一般相似问题，分类数等和问题紧密相关的output层与最后一个直接与output相关的隐层不可以直接用，仍需自己训练。
如下图所示，在已训练出一个复杂net后，迁移到相对简单的net时，hidden1和2固定不动，hidden3稍作变化，hidden4和output自己训练。。这在没有自己gpu情况下是非常节省时间的做法。
# 只选取需要的操作x=tf.get_default_graph().get_tensor_by_name(x:0)y=tf.get_default_graph().get_tensor_by_name(y:0)accuracy =tf.get_default_graph().get_tensor_by_name(eval/accuracy:0)training_op =tf.get_default_graph().get_operation_by_name(gradientdescent)# 如果你是原模型的作者，可以赋给模型一个清楚的名字保存下来forop in (x,y, accuracy, training_op): tf.add_to_collection(my_important_ops, op)# 如果你要使用这个模型x,y, accuracy, training_op =tf.get_collection(my_important_ops)# 训练时withtf.session()assess: saver.restore(sess,./my_model_final.ckpt) forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_new_model_final.ckpt) a. freezing the lower layers
训练时固定底层参数，达到freezing the lower layers的目的
# 以minist为例n_inputs=28*28# mnistn_hidden1=300# reusedn_hidden2=50# reusedn_hidden3=50# reusedn_hidden4=20# new!n_outputs=10# new!x= tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y= tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=tf.nn.relu, name=hidden1) # reused frozen hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name=hidden2) # reused frozen hidden2_stop =tf.stop_gradient(hidden2) hidden3 =tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu, name=hidden3) # reused, not frozen hidden4 =tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name=hidden4) # new! logits =tf.layers.dense(hidden4, n_outputs, name=outputs) # new!withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32), name=accuracy)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)reuse_vars =tf.get_collection(tf.graphkeys.global_variables, scope=hidden[123]) # regular expressionreuse_vars_dict = dict([(var.op.name, var)forvar in reuse_vars])restore_saver =tf.train.saver(reuse_vars_dict) #torestore layers1-3init =tf.global_variables_initializer()saver =tf.train.saver()withtf.session()assess: init.run() restore_saver.restore(sess,./my_model_final.ckpt) forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_new_model_final.ckpt) b. catching the frozen layers
训练时直接从lock层之后的层开始训练，catching the frozen layers
# 以minist为例n_inputs =28*28# mnistn_hidden1 =300# reusedn_hidden2 =50# reusedn_hidden3 =50# reusedn_hidden4 =20# new!n_outputs =10# new!x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=tf.nn.relu, name=hidden1) # reused frozen hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name=hidden2) # reused frozen & cached hidden2_stop =tf.stop_gradient(hidden2) hidden3 =tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu, name=hidden3) # reused, not frozen hidden4 =tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name=hidden4) # new! logits =tf.layers.dense(hidden4, n_outputs, name=outputs) # new!withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32), name=accuracy)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)reuse_vars =tf.get_collection(tf.graphkeys.global_variables, scope=hidden[123]) # regular expressionreuse_vars_dict = dict([(var.op.name, var)forvar in reuse_vars])restore_saver =tf.train.saver(reuse_vars_dict) #torestore layers1-3init =tf.global_variables_initializer()saver =tf.train.saver()importnumpyasnpn_batches = mnist.train.num_examples // batch_sizewithtf.session()assess: init.run() restore_saver.restore(sess,./my_model_final.ckpt) h2_cache = sess.run(hidden2, feed_dict={x: mnist.train.images}) h2_cache_test = sess.run(hidden2, feed_dict={x: mnist.test.images})# not shown in the book forepochinrange(n_epochs): shuffled_idx = np.random.permutation(mnist.train.num_examples) hidden2_batches = np.array_split(h2_cache[shuffled_idx], n_batches) y_batches = np.array_split(mnist.train.labels[shuffled_idx], n_batches) forhidden2_batch, y_batchinzip(hidden2_batches, y_batches): sess.run(training_op, feed_dict={hidden2:hidden2_batch, y:y_batch}) accuracy_val = accuracy.eval(feed_dict={hidden2: h2_cache_test,# not shown y: mnist.test.labels}) # not shown print(epoch,test accuracy:, accuracy_val) # not shown save_path = saver.save(sess,./my_new_model_final.ckpt)6、unsupervised pretraining 该方法的提出，让人们对深度学习网络的训练有了一个新的认识，可以利用不那么昂贵的未标注数据，训练数据时没有标注的数据先做一个pretraining训练出一个差不多的网络，再使用带label的数据做正式的训练进行反向传递，增进深度模型可用性
也可以在相似模型中做pretraining
7、faster optimizers 在传统的sgd上提出改进
有momentum optimization（最早提出，利用惯性冲量），nesterov accelerated gradient，adagrad（adaptive gradient每层下降不一样），rmsprop，adam optimization（结合adagrad和momentum，用的最多，是缺省的optimizer）
a. momentum optimization
记住之前算出的gradient方向，作为惯性加到当前梯度上。相当于下山时，sgd是静止的之判断当前最陡的是哪里，而momentum相当于在跑的过程中不断修正方向，显然更加有效。
b. nesterov accelerated gradient
只计算当前这点的梯度，超前一步，再往前跑一点计算会更准一些。
c. adagrad
各个维度计算梯度作为分母，加到当前梯度上，不同维度梯度下降不同。如下图所示，横轴比纵轴平缓很多，传统gradient仅仅单纯沿法线方向移动，而adagrad平缓的θ1走的慢点，陡的θ2走的快点，效果较好。
但也有一定缺陷，s不断积累，分母越来越大，可能导致最后走不动。
d. rmsprop（adadelta）
只加一部分，加一个衰减系数只选取相关的最近几步相关系数
e. adam optimization
目前用的最多效果最好的方法，结合adagrad和momentum的优点
# tensorflow中调用方法optimizer= tf.train.momentumoptimizer(learning_rate=learning_rate,momentum=0.9)optimizer= tf.train.momentumoptimizer(learning_rate=learning_rate,momentum=0.9, use_nesterov=true)optimizer= tf.train.rmspropoptimizer(learning_rate=learning_rate,momentum=0.9, decay=0.9, epsilon=1e-10)# 可以看出adamoptimizer最省心了optimizer= tf.train.adamoptimizer(learning_rate=learning_rate)8、learning rate scheduling learning rate的设置也很重要，如下图所示，太大不会收敛到全局最优，太小收敛效果最差。最理想情况是都一定情况缩小learning rate，先大后小
a. exponential scheduling
指数级下降学习率
initial_learning_rate=0.1decay_steps=10000decay_rate=1/10global_step= tf.variable(0, trainable=false)learning_rate= tf.train.exponential_decay(initial_learning_rate, global_step, decay_steps, decay_rate)optimizer= tf.train.momentumoptimizer(learning_rate, momentum=0.9)training_op= optimizer.minimize(loss, global_step=global_step)9、avoiding overfitting through regularization 解决深度模型过拟合问题
a. early stopping
训练集上错误率开始上升时停止
b. l1和l2正则化
# construct the neural networkbase_loss =tf.reduce_mean(xentropy, name=avg_xentropy)reg_losses =tf.reduce_sum(tf.abs(weights1)) +tf.reduce_sum(tf.abs(weights2))loss =tf.add(base_loss, scale * reg_losses, name=loss)with arg_scope( [fully_connected], weights_regularizer=tf.contrib.layers.l1_regularizer(scale=0.01)): hidden1 = fully_connected(x, n_hidden1, scope=hidden1) hidden2 = fully_connected(hidden1, n_hidden2, scope=hidden2) logits = fully_connected(hidden2, n_outputs, activation_fn=none,scope=out)reg_losses =tf.get_collection(tf.graphkeys.regularization_losses)loss =tf.add_n([base_loss] + reg_losses, name=loss) c. dropout
一种新的正则化方法，随机生成一个概率，大于某个阈值就扔掉，随机扔掉一些神经元节点，结果表明dropout很能解决过拟合问题。可强迫现有神经元不会集中太多特征，降低网络复杂度，鲁棒性增强。
加入dropout后，training和test的准确率会很接近，一定程度解决overfit问题
training =tf.placeholder_with_default(false, shape=(), name='training')dropout_rate =0.5# ==1- keep_probx_drop =tf.layers.dropout(x, dropout_rate, training=training)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x_drop, n_hidden1, activation=tf.nn.relu, name=hidden1) hidden1_drop =tf.layers.dropout(hidden1, dropout_rate, training=training) hidden2 =tf.layers.dense(hidden1_drop, n_hidden2, activation=tf.nn.relu, name=hidden2) hidden2_drop =tf.layers.dropout(hidden2, dropout_rate, training=training) logits =tf.layers.dense(hidden2_drop, n_outputs, name=outputs) d. max-norm regularization
可以把超出threshold的权重截取掉，一定程度上让网络更加稳定
defmax_norm_regularizer(threshold, axes=1, name=max_norm, collection=max_norm): defmax_norm(weights): clipped = tf.clip_by_norm(weights, clip_norm=threshold, axes=axes) clip_weights = tf.assign(weights, clipped, name=name) tf.add_to_collection(collection, clip_weights) returnnone# there is no regularization loss term returnmax_normmax_norm_reg = max_norm_regularizer(threshold=1.0)hidden1 = fully_connected(x, n_hidden1, scope=hidden1, weights_regularizer=max_norm_reg) e. date augmentation
深度学习网络是一个数据饥渴模型，需要很多的数据。扩大数据集，例如图片左右镜像翻转，随机截取，倾斜随机角度，变换敏感度，改变色调等方法，扩大数据量，减少overfit可能性
10、default dnn configuration
越南电信完全掌握5G技术，到2022年将用国产设备普及5G网络
小米电视增添视频通话功能，其它机型加紧适配中
无功补偿电容达不到额定电压值会产生什么后果？
DARPA人工智能作战应用研究将提升美军“算法战”能力
为了不被踢出AI的队伍，视觉深度模型都开始接私活了？
详解DNN训练中出现的问题与解决方法方法
远程桌面控制之跨网远程控制的方法
新型Proteus-RF任意波形发生器/收发机介绍
基于VF2的openWRT烧写、配置和测试
AUTO CAD2008快捷键命令大全
什么是高温线，高温线适用的两个关键因素是什么
LTpowerCAD设计工具的损耗估算和分解步骤的介绍
专家分享：器件的散热设计
日东科技携“IC贴合机”和“半导体回流焊”亮相合肥【IC China 2022】
dvteclipse软件的正确使用方法推荐
摩尔线程与OpenMMLab战略合作：推动算法框架与GPU协同发展，共筑AI开发者繁荣生态
曲面玻璃3d扫描设备线轮廓测量面轮廓检测间隙面差分析
苹果收购micro LED屏幕_首款使用OLED屏幕的设备
如何让电子工程师保持电路知识不遗忘？
三星推10纳米中端处理器Exynos 9610强化视觉深度处理
1、初始化权重 起初采用正态分布随机化初始权重，会使得原本单位的variance逐渐变得非常大。例如下图的sigmoid函数，靠近0点的梯度近似线性很敏感，但到了，即很强烈的输入产生木讷的输出。
采用xavier initialization，根据fan-in（输入神经元个数）和fan-out（输出神经元个数）设置权重。
并设计针对不同激活函数的初始化策略，如下图（左边是均态分布，右边正态分布较为常用）
2、激活函数 一般使用relu，但是不能有小于0的输入（dying relus）
a.leaky relu
改进方法leaky relu=max(αx,x)，小于0时保留一点微小特征。
具体应用
from tensorflow.examples.tutorials.mnist import input_datamnist = input_data.read_data_sets(/tmp/data/)reset_graph()n_inputs =28*28# mnistn_hidden1 =300n_hidden2 =100n_outputs =10x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=leaky_relu, name=hidden1) hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=leaky_relu, name=hidden2) logits =tf.layers.dense(hidden2, n_outputs, name=outputs)withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)learning_rate =0.01withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32))init =tf.global_variables_initializer()saver =tf.train.saver()n_epochs =40batch_size =50withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) ifepoch %5==0: acc_train = accuracy.eval(feed_dict={x: x_batch,y: y_batch}) acc_test = accuracy.eval(feed_dict={x: mnist.validation.images,y: mnist.validation.labels}) print(epoch,batch accuracy:, acc_train,validation accuracy:, acc_test) save_path = saver.save(sess,./my_model_final.ckpt) b. elu改进
另一种改进elu，在神经元小于0时采用指数变化
#just specify the activation function when building each layerx= tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)hidden1= tf.layers.dense(x, n_hidden1, activation=tf.nn.elu, name=hidden1) c. selu
最新提出的是selu（仅给出关键代码）
withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=selu, name=hidden1) hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=selu, name=hidden2) logits =tf.layers.dense(hidden2, n_outputs, name=outputs)# train 过程means = mnist.train.images.mean(axis=0, keepdims=true)stds = mnist.train.images.std(axis=0, keepdims=true) +1e-10withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) x_batch_scaled = (x_batch - means) / stds sess.run(training_op, feed_dict={x: x_batch_scaled,y: y_batch}) ifepoch %5==0: acc_train = accuracy.eval(feed_dict={x: x_batch_scaled,y: y_batch}) x_val_scaled = (mnist.validation.images - means) / stds acc_test = accuracy.eval(feed_dict={x: x_val_scaled,y: mnist.validation.labels}) print(epoch,batch accuracy:, acc_train,validation accuracy:, acc_test) save_path = saver.save(sess,./my_model_final_selu.ckpt)3、batch normalization 在2015年，有研究者提出，既然使用mini-batch进行操作，对每一批数据也可采用，在调用激活函数之前，先做一下normalization，使得输出数据有一个较好的形状，初始时，超参数scaling（γ）和shifting（β）进行适度缩放平移后传递给activation函数。步骤如下：
现今batch normalization已经被tensorflow实现成一个单独的层，直接调用
测试时，由于没有mini-batch，故训练时直接使用训练时的mean和standard deviation（），实现代码如下
import tensorflowastfn_inputs =28*28n_hidden1 =300n_hidden2 =100n_outputs =10batch_norm_momentum =0.9x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)training =tf.placeholder_with_default(false, shape=(), name='training')withtf.name_scope(dnn): he_init =tf.contrib.layers.variance_scaling_initializer() #相当于单独一层 my_batch_norm_layer = partial( tf.layers.batch_normalization, training=training, momentum=batch_norm_momentum) my_dense_layer = partial( tf.layers.dense, kernel_initializer=he_init) hidden1 = my_dense_layer(x, n_hidden1, name=hidden1) bn1 =tf.nn.elu(my_batch_norm_layer(hidden1))# 激活函数使用elu hidden2 = my_dense_layer(bn1, n_hidden2, name=hidden2) bn2 =tf.nn.elu(my_batch_norm_layer(hidden2)) logits_before_bn = my_dense_layer(bn2, n_outputs, name=outputs) logits = my_batch_norm_layer(logits_before_bn)# 输出层也做一个batch normalizationwithtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32)) init =tf.global_variables_initializer()saver =tf.train.saver()n_epochs =20batch_size =200#需要显示调用训练时得出的方差均值，需要额外调用这些算子extra_update_ops =tf.get_collection(tf.graphkeys.update_ops)#在training和testing时不一样withtf.session()assess: init.run() forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run([training_op, extra_update_ops], feed_dict={training:true,x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_model_final.ckpt)4、gradient clipp 处理gradient之后往后传，一定程度上解决梯度爆炸问题。（但由于有了batch normalization，此方法用的不多）
threshold =1.0optimizer = tf.train.gradientdescentoptimizer(learning_rate)grads_and_vars = optimizer.compute_gradients(loss)capped_gvs = [(tf.clip_by_value(grad, -threshold, threshold),var) forgrad,varingrads_and_vars]training_op = optimizer.apply_gradients(capped_gvs)5、重用之前训练过的层 （reusing pretrained layers）
对之前训练的模型稍加修改，节省时间，在深度模型训练（由于有很多层）中经常使用。
一般相似问题，分类数等和问题紧密相关的output层与最后一个直接与output相关的隐层不可以直接用，仍需自己训练。
如下图所示，在已训练出一个复杂net后，迁移到相对简单的net时，hidden1和2固定不动，hidden3稍作变化，hidden4和output自己训练。。这在没有自己gpu情况下是非常节省时间的做法。
# 只选取需要的操作x=tf.get_default_graph().get_tensor_by_name(x:0)y=tf.get_default_graph().get_tensor_by_name(y:0)accuracy =tf.get_default_graph().get_tensor_by_name(eval/accuracy:0)training_op =tf.get_default_graph().get_operation_by_name(gradientdescent)# 如果你是原模型的作者，可以赋给模型一个清楚的名字保存下来forop in (x,y, accuracy, training_op): tf.add_to_collection(my_important_ops, op)# 如果你要使用这个模型x,y, accuracy, training_op =tf.get_collection(my_important_ops)# 训练时withtf.session()assess: saver.restore(sess,./my_model_final.ckpt) forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_new_model_final.ckpt) a. freezing the lower layers
训练时固定底层参数，达到freezing the lower layers的目的
# 以minist为例n_inputs=28*28# mnistn_hidden1=300# reusedn_hidden2=50# reusedn_hidden3=50# reusedn_hidden4=20# new!n_outputs=10# new!x= tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y= tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=tf.nn.relu, name=hidden1) # reused frozen hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name=hidden2) # reused frozen hidden2_stop =tf.stop_gradient(hidden2) hidden3 =tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu, name=hidden3) # reused, not frozen hidden4 =tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name=hidden4) # new! logits =tf.layers.dense(hidden4, n_outputs, name=outputs) # new!withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32), name=accuracy)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)reuse_vars =tf.get_collection(tf.graphkeys.global_variables, scope=hidden[123]) # regular expressionreuse_vars_dict = dict([(var.op.name, var)forvar in reuse_vars])restore_saver =tf.train.saver(reuse_vars_dict) #torestore layers1-3init =tf.global_variables_initializer()saver =tf.train.saver()withtf.session()assess: init.run() restore_saver.restore(sess,./my_model_final.ckpt) forepoch inrange(n_epochs): foriteration inrange(mnist.train.num_examples // batch_size): x_batch, y_batch = mnist.train.next_batch(batch_size) sess.run(training_op, feed_dict={x: x_batch,y: y_batch}) accuracy_val = accuracy.eval(feed_dict={x: mnist.test.images, y: mnist.test.labels}) print(epoch,test accuracy:, accuracy_val) save_path = saver.save(sess,./my_new_model_final.ckpt) b. catching the frozen layers
训练时直接从lock层之后的层开始训练，catching the frozen layers
# 以minist为例n_inputs =28*28# mnistn_hidden1 =300# reusedn_hidden2 =50# reusedn_hidden3 =50# reusedn_hidden4 =20# new!n_outputs =10# new!x=tf.placeholder(tf.float32, shape=(none, n_inputs), name=x)y=tf.placeholder(tf.int64, shape=(none), name=y)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x, n_hidden1, activation=tf.nn.relu, name=hidden1) # reused frozen hidden2 =tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, name=hidden2) # reused frozen & cached hidden2_stop =tf.stop_gradient(hidden2) hidden3 =tf.layers.dense(hidden2_stop, n_hidden3, activation=tf.nn.relu, name=hidden3) # reused, not frozen hidden4 =tf.layers.dense(hidden3, n_hidden4, activation=tf.nn.relu, name=hidden4) # new! logits =tf.layers.dense(hidden4, n_outputs, name=outputs) # new!withtf.name_scope(loss): xentropy =tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits) loss =tf.reduce_mean(xentropy, name=loss)withtf.name_scope(eval): correct =tf.nn.in_top_k(logits,y,1) accuracy =tf.reduce_mean(tf.cast(correct,tf.float32), name=accuracy)withtf.name_scope(train): optimizer =tf.train.gradientdescentoptimizer(learning_rate) training_op = optimizer.minimize(loss)reuse_vars =tf.get_collection(tf.graphkeys.global_variables, scope=hidden[123]) # regular expressionreuse_vars_dict = dict([(var.op.name, var)forvar in reuse_vars])restore_saver =tf.train.saver(reuse_vars_dict) #torestore layers1-3init =tf.global_variables_initializer()saver =tf.train.saver()importnumpyasnpn_batches = mnist.train.num_examples // batch_sizewithtf.session()assess: init.run() restore_saver.restore(sess,./my_model_final.ckpt) h2_cache = sess.run(hidden2, feed_dict={x: mnist.train.images}) h2_cache_test = sess.run(hidden2, feed_dict={x: mnist.test.images})# not shown in the book forepochinrange(n_epochs): shuffled_idx = np.random.permutation(mnist.train.num_examples) hidden2_batches = np.array_split(h2_cache[shuffled_idx], n_batches) y_batches = np.array_split(mnist.train.labels[shuffled_idx], n_batches) forhidden2_batch, y_batchinzip(hidden2_batches, y_batches): sess.run(training_op, feed_dict={hidden2:hidden2_batch, y:y_batch}) accuracy_val = accuracy.eval(feed_dict={hidden2: h2_cache_test,# not shown y: mnist.test.labels}) # not shown print(epoch,test accuracy:, accuracy_val) # not shown save_path = saver.save(sess,./my_new_model_final.ckpt)6、unsupervised pretraining 该方法的提出，让人们对深度学习网络的训练有了一个新的认识，可以利用不那么昂贵的未标注数据，训练数据时没有标注的数据先做一个pretraining训练出一个差不多的网络，再使用带label的数据做正式的训练进行反向传递，增进深度模型可用性
也可以在相似模型中做pretraining
7、faster optimizers 在传统的sgd上提出改进
有momentum optimization（最早提出，利用惯性冲量），nesterov accelerated gradient，adagrad（adaptive gradient每层下降不一样），rmsprop，adam optimization（结合adagrad和momentum，用的最多，是缺省的optimizer）
a. momentum optimization
记住之前算出的gradient方向，作为惯性加到当前梯度上。相当于下山时，sgd是静止的之判断当前最陡的是哪里，而momentum相当于在跑的过程中不断修正方向，显然更加有效。
b. nesterov accelerated gradient
只计算当前这点的梯度，超前一步，再往前跑一点计算会更准一些。
c. adagrad
各个维度计算梯度作为分母，加到当前梯度上，不同维度梯度下降不同。如下图所示，横轴比纵轴平缓很多，传统gradient仅仅单纯沿法线方向移动，而adagrad平缓的θ1走的慢点，陡的θ2走的快点，效果较好。
但也有一定缺陷，s不断积累，分母越来越大，可能导致最后走不动。
d. rmsprop（adadelta）
只加一部分，加一个衰减系数只选取相关的最近几步相关系数
e. adam optimization
目前用的最多效果最好的方法，结合adagrad和momentum的优点
# tensorflow中调用方法optimizer= tf.train.momentumoptimizer(learning_rate=learning_rate,momentum=0.9)optimizer= tf.train.momentumoptimizer(learning_rate=learning_rate,momentum=0.9, use_nesterov=true)optimizer= tf.train.rmspropoptimizer(learning_rate=learning_rate,momentum=0.9, decay=0.9, epsilon=1e-10)# 可以看出adamoptimizer最省心了optimizer= tf.train.adamoptimizer(learning_rate=learning_rate)8、learning rate scheduling learning rate的设置也很重要，如下图所示，太大不会收敛到全局最优，太小收敛效果最差。最理想情况是都一定情况缩小learning rate，先大后小
a. exponential scheduling
指数级下降学习率
initial_learning_rate=0.1decay_steps=10000decay_rate=1/10global_step= tf.variable(0, trainable=false)learning_rate= tf.train.exponential_decay(initial_learning_rate, global_step, decay_steps, decay_rate)optimizer= tf.train.momentumoptimizer(learning_rate, momentum=0.9)training_op= optimizer.minimize(loss, global_step=global_step)9、avoiding overfitting through regularization 解决深度模型过拟合问题
a. early stopping
训练集上错误率开始上升时停止
b. l1和l2正则化
# construct the neural networkbase_loss =tf.reduce_mean(xentropy, name=avg_xentropy)reg_losses =tf.reduce_sum(tf.abs(weights1)) +tf.reduce_sum(tf.abs(weights2))loss =tf.add(base_loss, scale * reg_losses, name=loss)with arg_scope( [fully_connected], weights_regularizer=tf.contrib.layers.l1_regularizer(scale=0.01)): hidden1 = fully_connected(x, n_hidden1, scope=hidden1) hidden2 = fully_connected(hidden1, n_hidden2, scope=hidden2) logits = fully_connected(hidden2, n_outputs, activation_fn=none,scope=out)reg_losses =tf.get_collection(tf.graphkeys.regularization_losses)loss =tf.add_n([base_loss] + reg_losses, name=loss) c. dropout
一种新的正则化方法，随机生成一个概率，大于某个阈值就扔掉，随机扔掉一些神经元节点，结果表明dropout很能解决过拟合问题。可强迫现有神经元不会集中太多特征，降低网络复杂度，鲁棒性增强。
加入dropout后，training和test的准确率会很接近，一定程度解决overfit问题
training =tf.placeholder_with_default(false, shape=(), name='training')dropout_rate =0.5# ==1- keep_probx_drop =tf.layers.dropout(x, dropout_rate, training=training)withtf.name_scope(dnn): hidden1 =tf.layers.dense(x_drop, n_hidden1, activation=tf.nn.relu, name=hidden1) hidden1_drop =tf.layers.dropout(hidden1, dropout_rate, training=training) hidden2 =tf.layers.dense(hidden1_drop, n_hidden2, activation=tf.nn.relu, name=hidden2) hidden2_drop =tf.layers.dropout(hidden2, dropout_rate, training=training) logits =tf.layers.dense(hidden2_drop, n_outputs, name=outputs) d. max-norm regularization
可以把超出threshold的权重截取掉，一定程度上让网络更加稳定
defmax_norm_regularizer(threshold, axes=1, name=max_norm, collection=max_norm): defmax_norm(weights): clipped = tf.clip_by_norm(weights, clip_norm=threshold, axes=axes) clip_weights = tf.assign(weights, clipped, name=name) tf.add_to_collection(collection, clip_weights) returnnone# there is no regularization loss term returnmax_normmax_norm_reg = max_norm_regularizer(threshold=1.0)hidden1 = fully_connected(x, n_hidden1, scope=hidden1, weights_regularizer=max_norm_reg) e. date augmentation
深度学习网络是一个数据饥渴模型，需要很多的数据。扩大数据集，例如图片左右镜像翻转，随机截取，倾斜随机角度，变换敏感度，改变色调等方法，扩大数据量，减少overfit可能性
10、default dnn configuration
越南电信完全掌握5G技术，到2022年将用国产设备普及5G网络
小米电视增添视频通话功能，其它机型加紧适配中
无功补偿电容达不到额定电压值会产生什么后果？
DARPA人工智能作战应用研究将提升美军“算法战”能力
为了不被踢出AI的队伍，视觉深度模型都开始接私活了？
详解DNN训练中出现的问题与解决方法方法
远程桌面控制之跨网远程控制的方法
新型Proteus-RF任意波形发生器/收发机介绍
基于VF2的openWRT烧写、配置和测试
AUTO CAD2008快捷键命令大全
什么是高温线，高温线适用的两个关键因素是什么
LTpowerCAD设计工具的损耗估算和分解步骤的介绍
专家分享：器件的散热设计
日东科技携“IC贴合机”和“半导体回流焊”亮相合肥【IC China 2022】
dvteclipse软件的正确使用方法推荐
摩尔线程与OpenMMLab战略合作：推动算法框架与GPU协同发展，共筑AI开发者繁荣生态
曲面玻璃3d扫描设备线轮廓测量面轮廓检测间隙面差分析
苹果收购micro LED屏幕_首款使用OLED屏幕的设备
如何让电子工程师保持电路知识不遗忘？
三星推10纳米中端处理器Exynos 9610强化视觉深度处理