Byte range fails on GOES 16 URL present in tst_byterange.c

[abhibaruah]

To report a non-security related issue, please provide:

NetCDF version: 4.9.1
HDF5 version: 1.10.10
OS: Linux

I am trying to use netCDF byte range reading on the GOES16 URL which is mentioned in the file https://github.com/Unidata/netcdf-c/blob/main/nc_test/tst_byterange.c

"https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadC/2017/059/03/OR_ABI-L1b-RadC-M3C13_G16_s20170590337505_e20170590340289_c20170590340316.nc#mode=bytes"

However, nc_open fails with the status code -74.
Is this because of some changes to the netCDF code or changes in the GOES16 server?

#include <stdio.h>
#include <string.h>
#include <netcdf.h>
#include <iostream>

#define FILE_NAME "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadC/2017/059/03/OR_ABI-L1b-RadC-M3C13_G16_s20170590337505_e20170590340289_c20170590340316.nc#mode=bytes"

int
main()
{
    int status;
    int ncid;
    
    // open the NetCDF-4 file
    status = nc_open(FILE_NAME, NC_NOWRITE, &ncid);
    if(status != NC_NOERR) {
         std::cout << "Status: " << status << std::endl;
		 std::cout << "Could not open the file." << std::endl;
		 
    }
   
    // close the file 
    status = nc_close(ncid);
	
	std::cout << "Status after close: " << status << std::endl;

    return 0;
}

Status: -74
Could not open the file.
Status after close: -33

[DennisHeimbigner]

The -74 error code is NC_EDAPURL, which means a malformed URL was used.
If you insert that URL in a browser, does it work? It should ask you to save some data.

[abhibaruah]

If I paste the link on my browser, it downloads an nc file.
OR_ABI-L1b-RadC-M3C13_G16_s20170590337505_e20170590340289_c20170590340316.nc

nc_open on this downloaded file works.

[dopplershift]

@abhibaruah Can you try upgrading to netCDF 4.9.2? IIRC, there were some issues fixed in 4.9.1 around the byte range support. This URL worked for me just now with 4.9.2.

[abhibaruah]

@dopplershift Thanks for taking a look.
We have plans to upgrade to 4.9.2 soon.
I wanted to try it by using the prebuilt binaries (for 4.9.2) that you usually upload on your website. However, I could not find the ones for unix based OS. Do you know where I can download them from?

If they are not available, I will wait until we build netcdf 4.9.2 with our build harness and can verify the same.

[WardF]

Hi @abhibaruah, we only provide prebuilt libraries for Windows/Visual Studio. For all other platforms, we provide source code or the community-supported installers through the package manager specific to the distribution/platform.

[abhibaruah]

No problem Ward.
We will be upgrading to 4.9.2 soon and will let you know if it works for us or not.

[abhibaruah]

@dopplershift
With 4.9.2, the noaa-goes16 link works. Thanks for suggesting the upgrade.

However, I am trying another link:
"https://nrel-pds-wtk.s3.amazonaws.com/wtk-techno-economic/pywtk-data/met_data/0/0.nc#mode=bytes"
and with this link I get the open status as -74.

However, when I paste the link on my browser, it downloads an NC file.
Do you know why this link is failing?

[dopplershift]

I can confirm the failure with my setup as well. Not sure what's going on.

[DennisHeimbigner]

The problem is that this URL is using the default region, which is us-east-1.
https://nrel-pds-wtk.s3.amazonaws.com/wtk-techno-economic/pywtk-data/met_data/0/0.nc\#mode=bytes 0.nc.
But it turns out that the dataset is actually in us-west-2.
So, using this URL should work:
https://nrel-pds-wtk.s3.us-west-2.amazonaws.com/wtk-techno-economic/pywtk-data/met_data/0/0.nc\#mode=bytes 0.nc

[abhibaruah]

Confirmed that the corrected url works with 4.9.2.
Thanks Dennis for the solution.

[dopplershift]

The problem is that this URL is using the default region, which is us-east-1.

Is that because we're trying to parse the URL and use some S3 support directly? IMO the original URL should work just fine using our standard byte range support since it's a URL that supports HTTP Byte Range requests. At the very least, is there some way to fall back to that?

[DennisHeimbigner]

If you clear cache on your web browser and insert the original URL, you should see an error message
indicating a URL problem. That is how I learned that the dataset was in us-west-2.
Otherwise, I do not understand your comment. When you say, "...use some S3 support directly..."
can you elaborate on this?

[abhibaruah]

I tried to set the environment variable 'AWS_DEFAULT_REGION' to 'us-west-2'
% setenv AWS_REGION us-west-2
But the link
"https://nrel-pds-wtk.s3.amazonaws.com/wtk-techno-economic/pywtk-data/met_data/0/0.nc#mode=bytes"
still fails.
Shouldnt the AWS default region have changed to us-west-2 and this link work?

[dopplershift]

@DennisHeimbigner This works for me without error:

curl -v https://nrel-pds-wtk.s3.amazonaws.com/wtk-techno-economic/pywtk-data/met_data/0/0.nc > 0.nc

(also works with wget) so that's where I'm confused why the "default region" is at all relevant here.