5428ステップ後のオブジェクト検出のクラッシュ、TypeError： 'numpy.float64'オブジェクトは整数として解釈できません
私のオブジェクト検出器は複数回実行されましたが、この5428のマークでTypeErrorからクラッシュします
私はアナコンダで実行しています：
- numpy 1.18.1
- numpy-base 1.18.1
- tensorflow-gpu 1.14
以下のスニペットは、最も重要なエラーだと思いますか？：
2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
完全なトレースバックは以下のとおりです。
I0219 13:55:41.016854 15428 basic_session_run_hooks.py:260] loss = 0.0140173, step = 5400 (10.773 sec)
INFO:tensorflow:Saving checkpoints for 5428 into training/model.ckpt.
I0219 13:55:43.900022 15428 basic_session_run_hooks.py:606] Saving checkpoints for 5428 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0219 13:55:56.207441 15428 estimator.py:1145] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.009801 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.025418 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0219 13:55:58.025418 15428 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.573186 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.588815 15428 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0219 13:56:00.855241 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W0219 13:56:01.105266 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.
W0219 13:56:01.277014 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.
W0219 13:56:01.386395 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.
INFO:tensorflow:Done calling model_fn.
I0219 13:56:01.749697 15428 estimator.py:1147] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-02-19T13:56:01Z
I0219 13:56:01.781106 15428 evaluation.py:255] Starting evaluation at 2020-02-19T13:56:01Z
INFO:tensorflow:Graph was finalized.
I0219 13:56:02.489665 15428 monitored_session.py:240] Graph was finalized.
2020-02-19 13:56:02.508162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:06:00.0
2020-02-19 13:56:02.512995: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-02-19 13:56:02.516493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-02-19 13:56:02.518703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 Edge matrix:
2020-02-19 13:56:02.523922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-02-19 13:56:02.526614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-02-19 13:56:02.529223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8788 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0219 13:56:02.535778 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from training/model.ckpt-5428
I0219 13:56:02.538779 15428 saver.py:1280] Restoring parameters from training/model.ckpt-5428
INFO:tensorflow:Running local_init_op.
I0219 13:56:03.495252 15428 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0219 13:56:03.656017 15428 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 5 images.
I0219 13:56:06.852077 13368 coco_evaluation.py:205] Performing evaluation on 5 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0219 13:56:06.867704 13368 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0219 13:56:06.867704 13368 coco_tools.py:137] DONE (t=0.00s)
creating index...
index created!
2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
ret = func(*args)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
self._metrics = self.evaluate()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_tools.py", line 176, in __init__
iouType=iou_type)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
self.setDetParams()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
return fn(*args)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
(1) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
[[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\evaluation.py", line 272, in _evaluate_once
session.run(eval_ops, feed_dict)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1353, in run
raise six.reraise(*original_exc_info)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\six.py", line 703, in reraise
raise value
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1411, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1169, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[node IteratorGetNext (defined at model_main.py:105) ]]
(1) Out of range: End of sequence
[[node IteratorGetNext (defined at model_main.py:105) ]]
[[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'IteratorGetNext':
File "model_main.py", line 109, in <module>
tf.app.run()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "model_main.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
run_metadata=run_metadata))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
if self._save(run_context.session, global_step):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
if l.after_save(session, step):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
self._evaluate(global_step_value) # updates self.eval_result
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
self._evaluator.evaluate_and_export())
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
return _evaluate()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
self._call_model_fn_eval(input_fn, self.config))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1534, in _call_model_fn_eval
input_fn, ModeKeys.EVAL)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1022, in _get_features_and_labels_from_input_fn
self._call_input_fn(input_fn, mode))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\util.py", line 65, in parse_input_fn_result
result = iterator.get_next()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 426, in get_next
output_shapes=self._structure._flat_shapes, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1947, in iterator_get_next
output_shapes=output_shapes, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
op_def=op_def)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
ret = func(*args)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
self._metrics = self.evaluate()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_tools.py", line 176, in __init__
iouType=iou_type)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
self.params = Params(iouType=iouType) # parameters
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
self.setDetParams()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
File "<__array_function__ internals>", line 6, in linspace
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
[[node PyFunc_3 (defined at C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py:394) ]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'PyFunc_3':
File "model_main.py", line 109, in <module>
tf.app.run()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "model_main.py", line 105, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
saving_listeners)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
run_metadata=run_metadata)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
run_metadata=run_metadata))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
if self._save(run_context.session, global_step):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
if l.after_save(session, step):
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
self._evaluate(global_step_value) # updates self.eval_result
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
self._evaluator.evaluate_and_export())
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
hooks=self._eval_spec.hooks)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
return _evaluate()
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
self._call_model_fn_eval(input_fn, self.config))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1537, in _call_model_fn_eval
features, labels, ModeKeys.EVAL, config)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1146, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\model_lib.py", line 482, in model_fn
eval_config, list(category_index.values()), eval_dict)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\eval_util.py", line 947, in get_eval_metric_ops_for_evaluators
eval_dict))
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.Egg\object_detection\metrics\coco_evaluation.py", line 394, in get_estimator_eval_metric_ops
first_value_op = tf.py_func(first_value_func, [], tf.float32)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 480, in py_func
return py_func_common(func, inp, Tout, stateful, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 462, in py_func_common
func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 285, in _internal_py_func
input=inp, token=token, Tout=Tout, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_script_ops.py", line 159, in py_func
"PyFunc", input=input, token=token, Tout=Tout, name=name)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
op_def=op_def)
File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
これは、文字数制限のためにトレースバックの一部が欠落していますが、すべてnumpy TypeErrorのものです。
Numpyを1.17にダウングレードする必要があります。
また、gpuメモリの増加を制限することも役立つ可能性があることもわかりましたが、トレーニング速度が大幅に低下しました。