How to print the TP,FP,FN,TN in the terminal?

[ZwNSW]

❔Question

I want to get tp to analyze my own data set, but I cannot output it after running test.py？

Additional context

[github-actions]

Hello @ZwNSW, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@ZwNSW TP and FP vectors are computed here:

yolov5/utils/general.py

Lines 250 to 319 in c8c5ef3

	def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, fname='precision-recall_curve.png'):
	""" Compute the average precision, given the recall and precision curves.
	Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
	# Arguments
	tp: True positives (nparray, nx1 or nx10).
	conf: Objectness value from 0-1 (nparray).
	pred_cls: Predicted object classes (nparray).
	target_cls: True object classes (nparray).
	plot: Plot precision-recall curve at mAP@0.5
	fname: Plot filename
	# Returns
	The average precision as computed in py-faster-rcnn.
	"""
	# Sort by objectness
	i = np.argsort(-conf)
	tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
	# Find unique classes
	unique_classes = np.unique(target_cls)
	# Create Precision-Recall curve and compute AP for each class
	px, py = np.linspace(0, 1, 1000), [] # for plotting
	pr_score = 0.1 # score to evaluate P and R https://github.com/ultralytics/yolov3/issues/898
	s = [unique_classes.shape[0], tp.shape[1]] # number class, number iou thresholds (i.e. 10 for mAP0.5...0.95)
	ap, p, r = np.zeros(s), np.zeros(s), np.zeros(s)
	for ci, c in enumerate(unique_classes):
	i = pred_cls == c
	n_gt = (target_cls == c).sum() # Number of ground truth objects
	n_p = i.sum() # Number of predicted objects
	if n_p == 0 or n_gt == 0:
	continue
	else:
	# Accumulate FPs and TPs
	fpc = (1 - tp[i]).cumsum(0)
	tpc = tp[i].cumsum(0)
	# Recall
	recall = tpc / (n_gt + 1e-16) # recall curve
	r[ci] = np.interp(-pr_score, -conf[i], recall[:, 0]) # r at pr_score, negative x, xp because xp decreases
	# Precision
	precision = tpc / (tpc + fpc) # precision curve
	p[ci] = np.interp(-pr_score, -conf[i], precision[:, 0]) # p at pr_score
	# AP from recall-precision curve
	for j in range(tp.shape[1]):
	ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
	if j == 0:
	py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5
	# Compute F1 score (harmonic mean of precision and recall)
	f1 = 2 * p * r / (p + r + 1e-16)
	if plot:
	py = np.stack(py, axis=1)
	fig, ax = plt.subplots(1, 1, figsize=(5, 5))
	ax.plot(px, py, linewidth=0.5, color='grey') # plot(recall, precision)
	ax.plot(px, py.mean(1), linewidth=2, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())
	ax.set_xlabel('Recall')
	ax.set_ylabel('Precision')
	ax.set_xlim(0, 1)
	ax.set_ylim(0, 1)
	plt.legend()
	fig.tight_layout()
	fig.savefig(fname, dpi=200)
	return p, r, ap, f1, unique_classes.astype('int32')

[ZwNSW]

@glenn-jocher Thanks for your answer.When I run the code you gave, there was an error like this.
" ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
TypeError: cannot unpack non-iterable numpy.float64 object"
I guess this should be an error during the assignment. But I don't know how to correct it.

[ZwNSW]

When I print i, tpc, fpc, and their types to the terminal, I don't know which part is tpc, fpc, because the output of the terminal is as shown in the figure.I want to get tp, fp, fn for each category, and look forward to your answers and responses.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rita9410]

@ZwNSW did you manage to find out the values ​​of tp, fp and fn?

[glenn-jocher]

@rita9410 TP and FP vectors are computed here:

yolov5/utils/general.py

Lines 250 to 319 in c8c5ef3

	def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, fname='precision-recall_curve.png'):
	""" Compute the average precision, given the recall and precision curves.
	Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
	# Arguments
	tp: True positives (nparray, nx1 or nx10).
	conf: Objectness value from 0-1 (nparray).
	pred_cls: Predicted object classes (nparray).
	target_cls: True object classes (nparray).
	plot: Plot precision-recall curve at mAP@0.5
	fname: Plot filename
	# Returns
	The average precision as computed in py-faster-rcnn.
	"""
	# Sort by objectness
	i = np.argsort(-conf)
	tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
	# Find unique classes
	unique_classes = np.unique(target_cls)
	# Create Precision-Recall curve and compute AP for each class
	px, py = np.linspace(0, 1, 1000), [] # for plotting
	pr_score = 0.1 # score to evaluate P and R https://github.com/ultralytics/yolov3/issues/898
	s = [unique_classes.shape[0], tp.shape[1]] # number class, number iou thresholds (i.e. 10 for mAP0.5...0.95)
	ap, p, r = np.zeros(s), np.zeros(s), np.zeros(s)
	for ci, c in enumerate(unique_classes):
	i = pred_cls == c
	n_gt = (target_cls == c).sum() # Number of ground truth objects
	n_p = i.sum() # Number of predicted objects
	if n_p == 0 or n_gt == 0:
	continue
	else:
	# Accumulate FPs and TPs
	fpc = (1 - tp[i]).cumsum(0)
	tpc = tp[i].cumsum(0)
	# Recall
	recall = tpc / (n_gt + 1e-16) # recall curve
	r[ci] = np.interp(-pr_score, -conf[i], recall[:, 0]) # r at pr_score, negative x, xp because xp decreases
	# Precision
	precision = tpc / (tpc + fpc) # precision curve
	p[ci] = np.interp(-pr_score, -conf[i], precision[:, 0]) # p at pr_score
	# AP from recall-precision curve
	for j in range(tp.shape[1]):
	ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
	if j == 0:
	py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5
	# Compute F1 score (harmonic mean of precision and recall)
	f1 = 2 * p * r / (p + r + 1e-16)
	if plot:
	py = np.stack(py, axis=1)
	fig, ax = plt.subplots(1, 1, figsize=(5, 5))
	ax.plot(px, py, linewidth=0.5, color='grey') # plot(recall, precision)
	ax.plot(px, py.mean(1), linewidth=2, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())
	ax.set_xlabel('Recall')
	ax.set_ylabel('Precision')
	ax.set_xlim(0, 1)
	ax.set_ylim(0, 1)
	plt.legend()
	fig.tight_layout()
	fig.savefig(fname, dpi=200)
	return p, r, ap, f1, unique_classes.astype('int32')

[rita9410]

@glenn-jocher ok, but how can I print these variables by each class?

[glenn-jocher]

@rita9410 YOLOv5 TP and FP vectors are computed here:

yolov5/utils/general.py

Lines 250 to 319 in c8c5ef3

	def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, fname='precision-recall_curve.png'):
	""" Compute the average precision, given the recall and precision curves.
	Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
	# Arguments
	tp: True positives (nparray, nx1 or nx10).
	conf: Objectness value from 0-1 (nparray).
	pred_cls: Predicted object classes (nparray).
	target_cls: True object classes (nparray).
	plot: Plot precision-recall curve at mAP@0.5
	fname: Plot filename
	# Returns
	The average precision as computed in py-faster-rcnn.
	"""
	# Sort by objectness
	i = np.argsort(-conf)
	tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
	# Find unique classes
	unique_classes = np.unique(target_cls)
	# Create Precision-Recall curve and compute AP for each class
	px, py = np.linspace(0, 1, 1000), [] # for plotting
	pr_score = 0.1 # score to evaluate P and R https://github.com/ultralytics/yolov3/issues/898
	s = [unique_classes.shape[0], tp.shape[1]] # number class, number iou thresholds (i.e. 10 for mAP0.5...0.95)
	ap, p, r = np.zeros(s), np.zeros(s), np.zeros(s)
	for ci, c in enumerate(unique_classes):
	i = pred_cls == c
	n_gt = (target_cls == c).sum() # Number of ground truth objects
	n_p = i.sum() # Number of predicted objects
	if n_p == 0 or n_gt == 0:
	continue
	else:
	# Accumulate FPs and TPs
	fpc = (1 - tp[i]).cumsum(0)
	tpc = tp[i].cumsum(0)
	# Recall
	recall = tpc / (n_gt + 1e-16) # recall curve
	r[ci] = np.interp(-pr_score, -conf[i], recall[:, 0]) # r at pr_score, negative x, xp because xp decreases
	# Precision
	precision = tpc / (tpc + fpc) # precision curve
	p[ci] = np.interp(-pr_score, -conf[i], precision[:, 0]) # p at pr_score
	# AP from recall-precision curve
	for j in range(tp.shape[1]):
	ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
	if j == 0:
	py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5
	# Compute F1 score (harmonic mean of precision and recall)
	f1 = 2 * p * r / (p + r + 1e-16)
	if plot:
	py = np.stack(py, axis=1)
	fig, ax = plt.subplots(1, 1, figsize=(5, 5))
	ax.plot(px, py, linewidth=0.5, color='grey') # plot(recall, precision)
	ax.plot(px, py.mean(1), linewidth=2, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())
	ax.set_xlabel('Recall')
	ax.set_ylabel('Precision')
	ax.set_xlim(0, 1)
	ax.set_ylim(0, 1)
	plt.legend()
	fig.tight_layout()
	fig.savefig(fname, dpi=200)
	return p, r, ap, f1, unique_classes.astype('int32')

They don't print out by default, you'd have to introduce some custom code to see them.

fpc and tpc are the FP and TP arrays of shape (n,10), for the 10 iou thresholds of 0.5:0.95. If you look at the last row, this is the FP and TP count per iou threshold:

tpc.shape
Out[3]: (3444, 10)
fpc.shape
Out[4]: (3444, 10)
tpc[-1]
Out[5]: array([138, 124, 105,  91,  80,  66,  54,  38,  22,   9])
fpc[-1]
Out[6]: array([3306, 3320, 3339, 3353, 3364, 3378, 3390, 3406, 3422, 3435])

So at 0.5 iou and 0.001 confidence threshold, for class 0, dataset inference results in 138 TPs and 3306 FPs.

[dariogonle]

@rita9410 YOLOv5 TP and FP vectors are computed here:

yolov5/utils/general.py

Lines 250 to 319 in c8c5ef3

	def ap_per_class(tp, conf, pred_cls, target_cls, plot=False, fname='precision-recall_curve.png'):
	""" Compute the average precision, given the recall and precision curves.
	Source: https://github.com/rafaelpadilla/Object-Detection-Metrics.
	# Arguments
	tp: True positives (nparray, nx1 or nx10).
	conf: Objectness value from 0-1 (nparray).
	pred_cls: Predicted object classes (nparray).
	target_cls: True object classes (nparray).
	plot: Plot precision-recall curve at mAP@0.5
	fname: Plot filename
	# Returns
	The average precision as computed in py-faster-rcnn.
	"""
	# Sort by objectness
	i = np.argsort(-conf)
	tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]
	# Find unique classes
	unique_classes = np.unique(target_cls)
	# Create Precision-Recall curve and compute AP for each class
	px, py = np.linspace(0, 1, 1000), [] # for plotting
	pr_score = 0.1 # score to evaluate P and R https://github.com/ultralytics/yolov3/issues/898
	s = [unique_classes.shape[0], tp.shape[1]] # number class, number iou thresholds (i.e. 10 for mAP0.5...0.95)
	ap, p, r = np.zeros(s), np.zeros(s), np.zeros(s)
	for ci, c in enumerate(unique_classes):
	i = pred_cls == c
	n_gt = (target_cls == c).sum() # Number of ground truth objects
	n_p = i.sum() # Number of predicted objects
	if n_p == 0 or n_gt == 0:
	continue
	else:
	# Accumulate FPs and TPs
	fpc = (1 - tp[i]).cumsum(0)
	tpc = tp[i].cumsum(0)
	# Recall
	recall = tpc / (n_gt + 1e-16) # recall curve
	r[ci] = np.interp(-pr_score, -conf[i], recall[:, 0]) # r at pr_score, negative x, xp because xp decreases
	# Precision
	precision = tpc / (tpc + fpc) # precision curve
	p[ci] = np.interp(-pr_score, -conf[i], precision[:, 0]) # p at pr_score
	# AP from recall-precision curve
	for j in range(tp.shape[1]):
	ap[ci, j], mpre, mrec = compute_ap(recall[:, j], precision[:, j])
	if j == 0:
	py.append(np.interp(px, mrec, mpre)) # precision at mAP@0.5
	# Compute F1 score (harmonic mean of precision and recall)
	f1 = 2 * p * r / (p + r + 1e-16)
	if plot:
	py = np.stack(py, axis=1)
	fig, ax = plt.subplots(1, 1, figsize=(5, 5))
	ax.plot(px, py, linewidth=0.5, color='grey') # plot(recall, precision)
	ax.plot(px, py.mean(1), linewidth=2, color='blue', label='all classes %.3f mAP@0.5' % ap[:, 0].mean())
	ax.set_xlabel('Recall')
	ax.set_ylabel('Precision')
	ax.set_xlim(0, 1)
	ax.set_ylim(0, 1)
	plt.legend()
	fig.tight_layout()
	fig.savefig(fname, dpi=200)
	return p, r, ap, f1, unique_classes.astype('int32')

They don't print out by default, you'd have to introduce some custom code to see them.

fpc and tpc are the FP and TP arrays of shape (n,10), for the 10 iou thresholds of 0.5:0.95. If you look at the last row, this is the FP and TP count per iou threshold:

tpc.shape
Out[3]: (3444, 10)
fpc.shape
Out[4]: (3444, 10)
tpc[-1]
Out[5]: array([138, 124, 105,  91,  80,  66,  54,  38,  22,   9])
fpc[-1]
Out[6]: array([3306, 3320, 3339, 3353, 3364, 3378, 3390, 3406, 3422, 3435])

So at 0.5 iou and 0.001 confidence threshold, for class 0, dataset inference results in 138 TPs and 3306 FPs.

@glenn-jocher, is it a way to print the TPs and FPs at a certain confidence? I mean I'd like to know the number of TPs and FPs at 0.5 iou and 0.60 confidence. Is it possible?

And what does the 3444 means in the tcp.shape output?

[glenn-jocher]

@dariogonle the example you copied already prints them out at 10 different IoUs from 0.5 to 0.95. Results are evaluated at the --conf you supply.

[dariogonle]

@glenn-jocher thank you for your response, but I would like the following: When I do a test I get the following Precision and Recall.

I'd like to know the number of TP and FP used to calculate that Precision and Recall. The precision and recall given are for a certain confidence (the one that maximizes the F1), 0.75 in this case. When I run this test (default conf-thres = 0.001) I get the following TPs and FPs.

So the supposed precision, for iou=0.5, should be => P = 262/(262+1984) = 0.11, but in the output the precision is 0.89. If I print the n_l variable in ap_per_class I get 284, so the recall should be R = 262/284 = 0.92, but the output recall is 0.78.

I was wondering how recall and precision is calculated. I guess that the number of TPs and FPs shown in the second image are for all confiances, but the precision and recall shown in the first image is for a certain level of confidence (0.75 in this case, because it is the value that maximizes the F1). That's why I asked you if there is a way to output the TPs and FPs for a certain confidence.

I'm sure that I'm missing something, but I can't figure it out. Thank you in advance.