-
Notifications
You must be signed in to change notification settings - Fork 774
Description
There has been a known issue with random failures confirming APC LTL shipments that produced the following error during the "confirmTransportationOptions" (CTO) stage:
'operationProblems' => [
{
'message' => 'ERROR: Something went wrong. Please try again later.',
'severity' => 'ERROR',
'code' => 'InternalServerError'
}
This error would be random and clear -- but since Feb 28th, 100% of our attempts to ship APC LTL fail with this error (NOTE: APC UPS works no problem and we have successfully created APC LTL in the past, including Feb 25th). We have opened cases and given various "ideas/theories" of what could be wrong -- we believe we have ruled them all out. This is what we have been told and/or tested:
#1: "Transportation Options Timing Out" - There is a 20 minute time out on transportation options. We have made sure that start to finish on a plan is done well within this window. The error still occurs.
#2: "You need to set a delivery window" - Delivery windows are only required if a pre-condition flag is set, and it seems on APC LTL that this flag is never (or rarely?) set. Regardless, we have tested both setting a DW and not setting a DW and the error still occurs. (support also confirmed our transportation option did not require it). Also, when we test USE YOUR OWN CARRIER LTL, if we don't set a DW we get a very specific error message stating we must set a DW which we don't get with APC LTL.
#3: "getInboundOperationStatus is being called too quickly/often" - we have been told that if you call GIO too fast/repeatedly that it can cause random errors. We have eliminated this issue by waiting 20+ seconds after calling CTO and the error still occurs.
#4: That it could be related to certain SKUs or combinations - we have tested at least 15 combinations of SKUs including single SKUs this weekend - every single one failed at the CTO stage. We ARE able to ship via APC UPS for the same SKUs.
#5: "It was due to our contact settings in the plan/shipment" - We made sure that our contact information was identical in all steps that require it.
#6: That it has to due with a specific center or # of splits (5 splits or 1 split) - We have tested various single centers and various 5 center options, all failed with the same error.
#7: That maybe some combination of calls was confusing the system -- for example we were calling GTO on various placements within the plan to compare pricing, etc. We have avoided this issue by testing doing the limited # of calls to complete a shipment.
#8: That it has to do with a specific carrier -- we have tested many different carriers and all failed.
#9: We have tested USE YOUR OWN CARRIER - LTL as well as APC UPS and they work fine. So this is specifically related to APC LTL.
Interesting notes:
-
Is there something about Fridays/Weekends? While we have had random failures on week-days, on Friday Feb 21st in the afternoon we had failures and on Monday things worked fine. Then on Friday Feb 28, we had failures in the afternoon. Coincidence or is it something related to business days of availability?
-
When the CTO fails, it seems that you can't fully cancel the plan -- we can rename it but it stays in "Working" status on the Shipping Queue page.
-
It seems when CTO fails, the status of the shipment is kind of mixed. Seller Central says the shipment was created, yet still has a "Create Shipment" button -- and when you click "Create Shipment", it says we have not completed the 4 required steps in the API for the process to complete. It's kind of like the CTO partially completes but then fails in the middle -- leaving the plan/shipment in semi-complete status.
-
BEFORE we call CTO, we can call "generateTransportationOptions" (GTO) successfully, over and over and get updated quotes/options. However once the CTO failure happens, GTO gives a new error stating "pallet and freight info has not been provided". We are making the exact same call to GTO both before and after the CTO failure. This is the error received when calling GTO AFTER CTO fails. Basically once CTO fails it "ruins" the placement and the entire process has to be started over.
'operationProblems' => [
{
'message' => 'WARNING: Since pallet and freight info has not been provided, we may not be able to produce PCP-LTL options as part of GenerateTransportationOptions for the following shipment IDs: [sh-XXXX]',
'severity' => 'WARNING',
'code' => 'BadRequest'
},
{
'message' => 'ERROR: Sorry, something went wrong. Please try again, or refresh the page.',
'severity' => 'ERROR',
'code' => 'FBA_INB_0115'
}
Regardless of the cause, the most frustrating thing is that the error message is useless -- it provides no guidance or explanation of the problem. This issue has been reported months ago at this point and is still going on -- and seems to be getting worse.
I have also spent countless hours building complex debugging log tools so that I can provide detailed call logs to Amazon to troubleshoot... and countless hours actually running tests, which probably take 10+ minutes each of just sitting staring at a screen. I've probably sent Amazon 10-15 detailed logs of this issue at this point.
This is unacceptable, it's costing sellers significant money and needs to be resolved ASAP.