PIE applications using "optarg" do not work on newer gcc

[nyh]

Trying to run the memcached application (scripts/build image=memcached; scripts/run.py) no longer works. This is on the same Fedora 22 machine which used to run this correctly, so it appears to be a regression. The output shows a page fault in the kernel, with gdb reporting the following stack trace:

#8  <signal handler called>
#9  0x00000000004658fc in atoi (s=0x0) at musl/src/stdlib/atoi.c:7
#10 0x0000100000c16860 in main ()
#11 0x000000000040c863 in osv::application::run_main (
    this=this@entry=0xffffa00002df2710, path="/memcached", argc=argc@entry=5, 
    argv=argv@entry=0xffffa00002d91200) at core/app.cc:293
#12 0x000000000040f159 in osv::application::run_main (
    this=this@entry=0xffffa00002df2710) at core/app.cc:310
#13 0x000000000020c6f7 in osv::application::main (this=0xffffa00002df2710)
    at core/app.cc:238

So apparently, something we recently changed (in command line arguments? environment variables?) caused memcached's main() function (unfortunately, no line number information, but it's the main() function of memcached) to do an atoi on a null string.

To add insult to injury, I also see this:

$ scripts/run.py -e "memcached -u root"
OSv v0.24-21-g522f1da
eth0: 192.168.122.15
Memcached for OSV, based on Memcached 1.4.21
can't run as root without the -u switch

So something appears to be broken with the command line processing?

Also note I can reproduce the first problem (a crash) by using the command line

scripts/run.py -e "memcached -t 1"

(so it is not limited to the default command line).

[nyh]

The problem is caused by the application code gets optarg (see getopt(3)) as a null pointer.

It appears to not be a recent regression, bug might be related to the behavior of newer compilers, similar to what we fixed in commit 5430972

[nyh]

Huston, we have a problem :-(

This bug surfaced because of copy relocations are used by newer versions of gcc when compiling a PIE, as explained commit 5430972

The only place I found properly explaining what the right thing to do with copy relocations, and what they are really supposed to do, is here: http://www.shrubbery.net/solaris9ab/SUNWdev/LLM/p22.html#CHAPTER4-84604.

I believe the problem we're seeing is as follows:

The memcached application uses "extern char *optarg" (see getopt(3)). When gcc creates a PIE, it uses the copy relocation trick to save one indirection: It creates an area for this "optarg" variable in the the PIE, on startup it copies the initial value of this variable from the shared libraries (in our case, OSv), and from now on when either the executable or shared libraries (such as glibc) search for the "optarg" variable, they don't find the one in glibc, but rather find the one in the PIE, and all of them access it.

But in OSv, while the executable will indeed find its own "optarg", our glibc implementation isn't a shared library, it's a statically linked part of the kernel, and it will always use its own copy of optarg (determined during the static linking of the kernel). So the application and OSv work on separate copies of optarg, and the application won't see what the getopt() library function was trying to write to it.

We didn't see this problem in the test case of commit 5430972 because there we considered the "stdout" variable, which is never modified by OSv after its initialization (its contents is just an unchanging pointer to a certain object), so just copying the value once was sufficient - and we didn't notice that after this copying, the application and OSv were actually reading from two different copies of this pointer. The new problem with getopt() occurs because the application expects to continue to see changes that OSv is doing to this variable.

One quick workaround on how to get the "memcached" application quickly working again is not to compile memcached (and other applications with this problem) as PIE, and rather compile them as shared library :-(

A more complete solution must rememeber the following concern: OSv allows running multiple executables (in parallel or one after another), so every time getopt() is called, it might need to write optarg to a different place belonging to the currently running executable.

So perhaps we can do something like this:

• OSv code which uses "optarg" must search for it in the caller's ELF program (as usual, this searches first in the executable, and only lastly searches OSv itself).
• This search must be done every time getopt() is used, because perhaps the executable changed between calls
• However, this search should be cached to be more efficient in the usual case where the same elf::program is used and the executable did not change.
• We need to do this not only for optarg but for other things where OSv and the application communicate via a global variable. We should have a general function or macro make changing a few of these easy.
• Errno seems it should have been an obvious example of this problem (communication of application and osv through a global variable), but it isn't! Gcc is creates an "undefined" symbol for __errno_location, and doesn't use a copy relocation. Perhaps this is because errno is a per-thread variable, so it uses different mechnisms for dynamic linking (TLS)?
• It seems this problem is not common - the entire memcached has just two copy relocations (for optarg and stderr). But we need to figure out all the global variables which can cause it.

[nyh]

I changed in commit bdf7e9f the "memcached" app Makefile to compile as shared object instead of PIE, and work around this problem.
Of course we still need to fix this problem (and when we do, we can test it by changing apps/memcached/Makefile to switch back the commented lines to use PIE instead of shared object).

[wkozaczuk]

I am guessing this applies also to optind, opterr, optopt as specified in getopt description. Is the scope of this bug even bigger and it affects any app that is linked to use copy relocations?

I think many coreutils apps are affected by this bug and obviously any app that parses arguments using getopt() family of functions.

I also noticed that OSv is missing __progname and __progname_full which are referenced by many core utils. Adding something like this to OSv make them work (minus optarg problem):

const char* __progname = "bla";
const char* __progname_full = __progname;

__progname is also reported using copy relocation so it can be fixed same way as stderr. The real solution for __progname should somehow copy file portion of argv[0] and __progname_full copy of argv[0]. I have found out that __progname are part of some kind GNU C extension and originally comes from BSD - https://rosettacode.org/wiki/Program_name#C.

Would the solution to fix be simpler if for now we limited it to single executable scenario (then sequential)? I am not sure parallel case is even a priority.

[nyh]

@wkozaczuk yes, "optarg" was the only problem in memcached, but the other global variables between OSv and the applications will also have the same problem - they are just rarer. I suggested above how OSv can solve this with a function to find the "right" optarg, and the same function can be used to find the right "optind" and perhaps other such variables.

The __progname is a separate issue, can you please open one? I'm not sure it's related to copy relocation, does it have a copy relocation? Also is it special to non-PIE compilation? The core utils, when compiled as PIE, don't need __progname?

[wkozaczuk]

@nyh yes __progname does have copy relocation so it is related but maybe different on its one.