isatty() does not return correct value in cygwin and msys2.

[tyan0]

Overview

In cygwin (3.0.7 or older) and msys2, isatty() does not return correct value.

Description

In cygwin 3.0.7 or older and msys2, return value of isatty() is not correct. The value returned is almost opposite. It returns 0 for pty and returns 1 when it is piped or redirected to /dev/null or /dev/zero. These results are exact opposite to expected values. The correct case is only when it is redirected to normal file. In this case, isatty() returns 0 as expected.

In cygwin 3.1.0 and later, it returns correct value 1 for pty, however, also returns 1 for pipe, /dev/null and /dev/zero.

The Cause

The rc setting at the bottom of jansi_isatty.c is completly reversed. Moreover, checking FILE_TYPE_CHAR is not enough because GetFileType() returns FILE_TYPE_CHAR for cygwin /dev/null and /dev/zero.

Patch for this issue

diff --git a/src/main/native-package/src/jansi_isatty.c b/src/main/native-package/src/jansi_isatty.c
index 26f576c..24745b4 100644
--- a/src/main/native-package/src/jansi_isatty.c
+++ b/src/main/native-package/src/jansi_isatty.c
@@ -50,12 +50,13 @@ JNIEXPORT jint JNICALL CLibrary_NATIVE(isatty)
 	BYTE buffer[1024];
 	PFILE_NAME_INFORMATION nameinfo = (PFILE_NAME_INFORMATION) buffer;
 	PWSTR name;
+	DWORD mode;
 
 	CLibrary_NATIVE_ENTER(env, that, CLibrary_isatty_FUNC);
 	/* check if fd is a pipe */
 	HANDLE h = (HANDLE) _get_osfhandle(arg0);
 	DWORD t = GetFileType(h);
-	if (t == FILE_TYPE_CHAR) {
+	if (t == FILE_TYPE_CHAR && GetConsoleMode(h, &mode)) {
 		rc = 1;
 	}
 	else if (t != FILE_TYPE_PIPE) {
@@ -90,9 +91,9 @@ JNIEXPORT jint JNICALL CLibrary_NATIVE(isatty)
 				 */
 				if ((!wcsstr(name, L"msys-") && !wcsstr(name, L"cygwin-"))
 						|| !wcsstr(name, L"-pty")) {
-					rc = 1;
-				} else {
 					rc = 0;
+				} else {
+					rc = 1;
 				}
 			}
 		}