When using stdout output, let's avoid any temporary files

[d0vgan]

I have a general suggestion to ctags regarding the behavior that have probably been being here since the origin of ctags.
When ctags uses stdout output (when -f - command line option is provided), it anyway creates a temporary file during the tags generation.
To me, such behavior ruins the whole idea of the stdout output. Personally I was using -f - with the only purpose to avoid any temporary file creation. My expectations regarding -f - were the following:

1. ctags is generating its output tags in a memory
2. ctags does not use any intermediate temporary file during this
3. ctags outputs the generated tags from the memory to stdout.

However, in reality ctags generates a temporary tags file that seems absolutely useless in this scenario since the output goes to stdout.
In todays world of SSD disks (with their limited write cycles), I'd like to avoid temporary files as much as possible.
Please consider making -f - to not use any temporary file.

[masatake]

Sorry, but I will add more code using more temporary files in the future for implementing multi-file multip-pass tagging (#2741 and #1495). readtags command also uses a temporary file for streaming processing.

Sending pull requests to avoid using temporary files is welcome.
If you will not do, please, mount tmpfs to /tmp after attaching enough RAM to your PC to extend the life cycle of your SSD.
Here I expect you runs ctags on Linux.

[masatake]

[jet@living]~/var/ctags-github% find -type f | xargs cat | strace -e open,openat sort > /dev/null
...
openat(AT_FDCWD, "/tmp/sortUrWWVA", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
openat(AT_FDCWD, "/tmp/sortCqUFeR", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
openat(AT_FDCWD, "/tmp/sortXljFwu", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
openat(AT_FDCWD, "/tmp/sort3j9b0v", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
...

Even if you don't use ctags, mounting tmpfs (with enough RAM) may extend the life of your SSD.

[d0vgan]

What is the underlying reason for this? I mean, why are temporary files required? Isn't all the logic done in memory?

[masatake]

What is the underlying reason for this?

Finding the reason is the first step to make a pull request.
ctags may help you to understand ctags:-)

[masatake]

Those who are interested in working on this topic, I wrote what I know.

Look at createTagsWithFallback1() in parse.c.
To handle #if/#ifdef/#else/#endif better, our c/c++ parser scans the same source code multiple times. In nth trial, the parser slows away the tags output of n-1th scanning.
The file-seeking is used in the process of the 'throwing away'.

[d0vgan]

I'd expect that the information that is written to and read from a tag file, may be in the simiilar way be written to and read from a memory? It might be easier to do in C++ by introducing a proper interface that abstracts the read/write operations from physical files (so we could substitute e.g. CMemoryFile instance instead of CPhysicalFile instance); but I believe the similar goal can be achieved in plain C by using structures and pointers to functions.
Given that, maybe it will make sense to implement the multi-file multi-pass tagging by means of "memory files" instead of physical files? Where a "memory file" might be represented by something like:

typedef struct sMemoryFile {
    unsigned char* pMem; // pointer to the allocated memory
    size_t nMemSize; // size of allocated memory
    size_t nPos; // the current position within a "file"
} tMemoryFile;

tMemoryFile* tMemoryFile_Create()
{
  tMemoryFile* pFile = (tMemoryFile*) malloc(sizeof(tMemoryFile));
  memset(pFile, 0, sizeof(tMemoryFile));
  return pFile;
}

void tMemoryFile_Close(tMemoryFile* pFile)
{
  if (pFile->pMem)
    free(pFile->pMem);
  memset(pFile, 0, sizeof(tMemoryFile));
}

size_t tMemoryFile_Read(tMemoryFile* pFile, unsigned char* pBuf, size_t nBytes)
{
  if (nBytes > pFile->nMemSize - pFile->nPos)
    nBytes = pFile->nMemSize - pFile->nPos;
  memcpy(pBuf, pFile->pMem + pFile->nPos, nBytes);
  return nBytes;
}

size_t tMemoryFile_Write(tMemoryFile* pFile, const unsigned char* pBuf, size_t nBytes)
{
  if (pFile->nPos + nBytes > pFile->nMemSize)
    tMemoryFile_Resize(pFile, nBytes); // allocates more memory
  memcpy(pFile->pMem + pFile->nPos, pBuf, nBytes);
  return nBytes;
}

As you can see, the similar "wrapper" can be written around the standard fopen, fwrite, fread and so on by introducing e.g. tPhysicalFile_Open, tPhysicalFile_Write, tPhysicalFile_Read and so on. And then you'll be able to use either tMemoryFile or tPhysicalFile depending on your needs in each particular situation.

[masatake]

We already have such a wrapper called Mio derived from Geany IDE.
https://github.com/universal-ctags/ctags/blob/master/main/mio.h