fastText stuck in while ((c = sb.sbumpc()) != EOF)

[mbyzhang]

When training a small dataset on Raspberry Pi, fastText uses more and more memory and finally "bad_alloc".
Debugging with GDB, I found fastText stuck in a loop which never terminates.

(gdb) n
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 
195	  while ((c = sb.sbumpc()) != EOF) {
(gdb) 
196	    if (c == ' ' || c == '\n' || c == '\r' || c == '\t' || c == '\v' ||
(gdb) 
197	        c == '\f' || c == '\0') {
(gdb) 
210	    word.push_back(c);
(gdb) 

[mbyzhang]

By adding #define EOF 0xff, everything works fine.
But I don't know what is happening behind it...

[mbyzhang]

After doing some research, I found char under ARM environment is unsigned char by default, which makes an char variable under ARM impossible to be -1 (EOF), and it underflows to 255.

So forcing char c; on line 192 in dictionary.cc to signed char c; just works.

I will submit a pull request later.

[EdouardGrave]

Hi @mbyzhang,

Thank you for reporting and fixing this issue!

Best,
Edouard.