Anyone on Mavericks with a Hugo compiler?

After upgrading to OS X Mavericks the Hugo compiler has done something strange. Text in the text bank is compiled slightly wrong. Occasionally a part of the text is shifted one character left. For example:

routine main { "one two three four five six seven eight nine ten" }
The output is “one two three four five six seveneight nine teen”. It doesn’t happen for everything in the text bank but still frequently, there’s maybe a 50/50 chance of it occurring in a longer text. I haven’t figured out any pattern to it. I’ve checked the resulting story file and it’s definitely the compiler that’s creating the file wrong, not the interpreter reading it wrong. Recompiling the compiler doesn’t help.

This might be a long shot, but does anyone have the Hugo compiler on OS X Mavericks and could test if it’s happening to them as well? Or would be willing to install it, which involves installing Xcode and compiling from the Unix sources.

Confirmed. I built v3.1.03 on my Mac and I get that result.

If I remove the “-O2” switch, the problem goes away. (It’s still buggy with “-O”.) My conclusion is that the Hugo compiler contains a bug which is being tweaked by new clang optimizations. It’s also possible that this is a clang (cc) bug, but clang is more widely-tested than Hugo is, I’m afraid.

I tried turning on the static analyzer. Long pedantic warning list follows:

Keep in mind that some of these are conditional. E.g. a warning “Assigned value is garbage or undefined” means that clang believes it’s logically possible to reach that statement with the value undefined. Sometimes it’s overly cautious.

However, I see some real bugs here. “free(objattr)” is bad, because objattr is a static array. There are a lot of mis-typed mallocs – “object_hash” is an array of ints, but its malloc is MAXOBJECTS*sizeof(int *). Probably not a problem in real life, but…

/Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile hc
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hc.c
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcbuild.c
source/hcbuild.c:535:5: warning: Value stored to ‘notflag’ is never read
notflag = false;
^ ~~~~~
source/hcbuild.c:543:6: warning: Value stored to ‘notflag’ is never read
notflag = true;
^ ~~~~
2 warnings generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hccode.c
source/hccode.c:65:2: warning: Value stored to ‘loopptr’ is never read
loopptr = codeptr;
^ ~~~~~~~
source/hccode.c:669:5: warning: Value stored to ‘eol’ is never read
eol = 1;
^ ~
source/hccode.c:790:7: warning: Value stored to ‘lastt’ is never read
lastt = COMMA_T;
^ ~~~~~~~
source/hccode.c:791:7: warning: Value stored to ‘evalue’ is never read
evalue = i;}
^ ~
source/hccode.c:986:6: warning: Value stored to ‘compbrackets’ is never read
compbrackets = brackets;
^ ~~~~~~~~
5 warnings generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcdef.c
source/hcdef.c:91:4: warning: Value stored to ‘t’ is never read
t = IDWord(wordnum);
^ ~~~~~~~~~~~~~~~
source/hcdef.c:526:11: warning: Assigned value is garbage or undefined
word[2] = tempword[1];
^ ~~~~~~~~~~~
source/hcdef.c:548:16: warning: Assigned value is garbage or undefined
word[words] = tempword[3];
^ ~~~~~~~~~~~
source/hcdef.c:552:19: warning: Assigned value is garbage or undefined
word[++words] = tempword[4];
^ ~~~~~~~~~~~
4 warnings generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcfile.c
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hclink.c
source/hclink.c:353:15: warning: Result of ‘malloc’ is converted to a pointer of type ‘unsigned char’, which is incompatible with sizeof operand type ‘char’
if ((cbuf = malloc(ccount * sizeof(char)))!=NULL)
^~~~~~ ~~~~~~~~~~~~
1 warning generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcmisc.c
source/hcmisc.c:399:3: warning: Value stored to ‘flag’ is never read
flag = 1;
^ ~
source/hcmisc.c:557:2: warning: Null pointer argument in call to string copy function
strcpy(s, a);
^~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/secure/_string.h:88:6: note: expanded from macro ‘strcpy’
? __builtin___strcpy_chk (dest, src, __darwin_obsz (dest))
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
source/hcmisc.c:557:2: warning: Null pointer argument in call to string copy function
strcpy(s, a);
^~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/secure/_string.h:89:6: note: expanded from macro ‘strcpy’
: __inline_strcpy_chk (dest, src))
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
source/hcmisc.c:1628:23: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(object_hash = malloc(MAXOBJECTS
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1647:25: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(property_hash = malloc(MAXPROPERTIES
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1678:22: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(alias_hash = malloc(MAXALIASES
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1687:22: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(array_hash = malloc(MAXARRAYS
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1697:25: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(constant_hash = malloc(MAXCONSTANTS
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1714:24: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(routine_hash = malloc(MAXROUTINES
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
source/hcmisc.c:1724:22: warning: Result of ‘malloc’ is converted to a pointer of type ‘int’, which is incompatible with sizeof operand type 'int
if (!(label_hash = malloc(MAXLABELS
sizeof(int *)))) goto MemoryError;
^~~~~~ ~~~~~~~~~~~~~
10 warnings generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hccomp.c
source/hccomp.c:352:4: warning: Argument to free() is the address of the global variable ‘objattr’, which is not memory allocated by malloc()
free(objattr);
^~~~~~~~~~~~~
1 warning generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcpass.c
source/hcpass.c:437:7: warning: Value stored to ‘flag’ is never read
flag = 1;
^ ~
1 warning generated.
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/hcres.c
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c source/stringfn.c
cc -I/usr/local/include -Isource -DDO_COLOR -O1 --analyze -DNCURSES -DGCC_UNIX -DUSE_TEMPFILES -c gcc/hcgcc.c

Ok, thanks. If I remove the -O2 flag it compiles but running it gives an “Abort trap: 6” error.

Huh, I didn’t get that. (On your three-line test source code.)

If you can get a crash, you can start attacking the problem with GDB, I guess.

(I tried a debugging malloc package, and it didn’t report anything. Are you testing with the test program you quoted, or a longer one?)

It crashes with the same error with all code, so does the interpreter. (To clarify, the C source compiles but the resulting Hugo compiler and interpreter are broken.)

I don’t know what the difference between our setups is, then.

This is Xcode 5.0 (5A1413), clang-500.2.76, which is not the absolute latest version. I can try upgrading, but at best this would let me start debugging Hugo source code, which is not really what I need to be working on these days. :slight_smile:

I’m using the stock Makefile with CFLAGS=-Wall, if that helps. In this configuration I can compile the test code and it executes correctly.

I have Xcode 5.0.1 5A2053. Otherwise the same.

It’s not a big deal for me, but hopefully someone finds a solution because others are bound to face the same problem in the future.

Yeah, upgraded to Xcode 5.0.1 (5A2053) (clang-500.2.79) and now I see the crash too.

Stack trace:

[...strcpy internals...]
#8  0x0000000100014ab8 in StripQuotes (a=0x100023300 "one two three four five six seven eight nine ten") at source/hcmisc.c:1884
#9  0x00000001000079a8 in IDWord (a=1) at source/hccode.c:1875
#10 0x00000001000037c6 in CodeLine () at source/hccode.c:440
#11 0x0000000100000feb in BuildCode (from=0 '\0') at source/hcbuild.c:230
#12 0x00000001000030b2 in BuildRoutine () at source/hcbuild.c:918
#13 0x0000000100017990 in Pass2 () at source/hcpass.c:375
#14 0x0000000100000800 in main (argc=2, argv=0x7fff5fbffa20) at source/hc.c:68

The fatal line is “strcpy(a, a+1)” where a is a pointer to the text string “one two three four five six seven eight nine ten”. strcpy() is documented as “do not call this on overlapping strings, you idiot” so I’m betting that’s the problem.

I tried replacing that line with “memmove(a, a+1, strlen(a+1))” which is safe (as far as I know). I was then able to compile test.hugo successfully. Then I couldn’t execute the resulting test.hex file, because he crashed – looks like the same problem, in hugo_splitpath().

Conclusion: someone needs to go through all the Hugo source code, search for this strcpy idiom, and fix it everywhere. (Or maybe #define a strcpy() macro that uses memmove() underneath.)

Just adding my notes as I work through this…

Checking my code out here:
hg.0branch.com/hugo-unix

I see the problem on these lines. There may be more, but it definitely seems to be here:

hcgcc.c line 74
hcmisc.c line 1884
stringfn.c line 109
heblank.c line 154