Keras：Tensorflow的黄金标准

苏神，问下，你的bert4keras支持徐亮的albert_tiny吗

感谢苏神总结，非常受用

苏神，看了这篇文章和bert4keras上对roberta的pretraining后，我在按照你的写法，改写task_seq2seq为单机多卡训练。但是一直报下面的错误：

Traceback (most recent call last):
File "/home/dell/anaconda3/envs/gpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds. for 'loss/seq2seq_loss_loss/strided_slice' (op: 'StridedSlice') with input shapes: [0], [1], [1], [1] and with computed input tensors: input[1] = , input[2] = , input[3] = .

我模型构建部分的代码是：
def build_multi_gpu_model():
"""
rewrite loss function to a Lambda layer;
using tf.keras and tf.distributed
"""
Lambda = keras.layers.Lambda
_model = keras.models.Model

bert = build_bert_model(
config_path,
application='seq2seq',
keep_words=keep_words, # 只保留keep_words中的字，精简原字表
return_keras_model=False
)

bert_model = bert.model
proba = bert_model.output

def seq2seq_loss(inputs):
y_in, y_pred, y_mask = inputs
y_in = y_in[:, 1:]
y_mask = y_mask[:, 1:]
y_pred = y_pred[:, :-1]

cross_entropy = K.sparse_categorical_crossentropy(y_in, y_pred)
cross_entropy = K.sum(cross_entropy * y_mask) / K.sum(y_mask)
return cross_entropy

loss = Lambda(seq2seq_loss, name='seq2seq_loss')([bert_model.inputs[0], proba, bert_model.inputs[1]])

train_model = _model(bert_model.inputs, loss)

# config optimizer
opt = extend_with_weight_decay(Adam)
if which_optimizer == 'lamb':
opt = extend_with_layer_adaptation(opt)
opt = extend_with_piecewise_linear_lr(opt)
opt_params = {
'learning_rate': learning_rate,
'lr_schedule': lr_schedule,
'weight_decay_rate': weight_decay_rate,
'exclude_from_weight_decay': exclude_from_weight_decay,
'bias_correction': False,
}
if grad_accum_steps > 1:
opt = extend_with_gradient_accumulation(opt)
opt_params['grad_accum_steps'] = grad_accum_steps

optimizer = opt(**opt_params)

# compile final model for multi GPU/TPU training
train_model.compile(
loss={
'seq2seq_loss': lambda y_true, y_pred: y_pred,
},
optimizer=optimizer,
)

# must pass checkpoint_path here, otherwise error occurs
if checkpoint_path is not None:
bert.load_weights_from_checkpoint(checkpoint_path)
else:
pass
return train_model

训练的时候：
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = build_multi_gpu_model()
model.summary()
model.fit_generator(
data_generator(),
steps_per_epoch=steps_per_epoch,
epochs=epochs,
callbacks=[evaluator, checkpoint, csv_logger]
)

想请教一下苏神，是我模型构建部分出错了么？还是多GPU的时候不能用model.fit_generator?

感谢苏神，我按照你说的，给generator中yield出来的inputs换成字典，key是模型输入层的name,但是还是报了一样的错误：
Traceback (most recent call last):
File "/home/dell/anaconda3/envs/gpu/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds. for 'loss/seq2seq_loss_loss/strided_slice' (op: 'StridedSlice') with input shapes: [0], [1], [1], [1] and with computed input tensors: input[1] = , input[2] = , input[3] = .
模型构建部分：
Lambda = keras.layers.Lambda
Input = keras.layers.Input
_model = keras.models.Model

bert = build_bert_model(
config_path,
application='seq2seq',
keep_words=keep_words, # 只保留keep_words中的字，精简原字表
return_keras_model=False
)

bert_model = bert.model
proba = bert_model.output

token_ids = Input(shape=(None,), dtype='float32', name='token_ids') # 目标id
is_masked = Input(shape=(None,), dtype='float32', name='is_masked') # mask标记

def seq2seq_loss(inputs):
y_in, y_pred, y_mask = inputs
y_in = y_in[:, 1:]
y_mask = y_mask[:, 1:]
y_pred = y_pred[:, :-1]

cross_entropy = K.sparse_categorical_crossentropy(y_in, y_pred)
cross_entropy = K.sum(cross_entropy * y_mask) / K.sum(y_mask)
return cross_entropy

loss = Lambda(seq2seq_loss, name='seq2seq_loss')([token_ids, proba, is_masked])

train_model = _model(bert_model.inputs + [token_ids, is_masked], loss)

# config optimizer
opt = extend_with_weight_decay(Adam)
if which_optimizer == 'lamb':
opt = extend_with_layer_adaptation(opt)
opt = extend_with_piecewise_linear_lr(opt)
opt_params = {
'learning_rate': learning_rate,
'lr_schedule': lr_schedule,
'weight_decay_rate': weight_decay_rate,
'exclude_from_weight_decay': exclude_from_weight_decay,
'bias_correction': False,
}
if grad_accum_steps > 1:
opt = extend_with_gradient_accumulation(opt)
opt_params['grad_accum_steps'] = grad_accum_steps

optimizer = opt(**opt_params)

# compile final model for multi GPU/TPU training
train_model.compile(
loss={
'seq2seq_loss': lambda y_true, y_pred: y_pred,
},
optimizer=optimizer,
)
是不是我的loss层有问题呢？谢谢苏神！
注：评论里怎么插入代码块呀，我用latex的\begin{lstlisting}[language=Python]\end{lstlisting}模块好像在预览里不起作用。。。

接上面的。。。我的data_generator长这样：
\begin{lstlisting}[language=Python]
def data_generator():
while True:
x_list, s_list = [], []
for a, b in read_texts():
x, s = tokenizer.encode(a, second_text=b, max_length=512)
x_list.append(x)
s_list.append(s)
if len(x) == batch_size:
x_list = sequence_padding(x_list)
s_list = sequence_padding(s_list)
inputs = {
'Input-Token': x_list,
'Input-Segment': s_list,
'token_ids': x_list,
'is_masked': s_list,
}
yield inputs, None
x_list, s_list = [], []
\end{lstlisting}@schwarz|comment-12510

在跑bert4keras demo时候 报这个错怎么解决呀tensorflow.python.framework.errors_impl.NotFoundError: Key bert/pooler/dense/kernel not found in checkpoint

在跑tf2.0 keras模型，fit的时候，跑到training_eager.py中，line 127 ：outs = model(inputs, **kwargs)，这个就是实际计算loss的时候的y_pred,但这个y_pred没有_keras_history,最开始compile的时候y_pred是有的，之前tf1.0也是有的，不知道是我使用的问题还是tf2.0更新的问题，他的base_layer.py也加了好多有关keras_history的东西，苏神有注意这个地方吗，困扰我好久了