Replace Makefile/make.bat with a Python script?

[brechtm]

A Python script would make it easier to support the various platforms (#3145, #3194). It would also mean that it can be used in the same way on all platforms, which is handy in a tox.ini, for example.

[brechtm]

Here is a (work-in-progress) script I wrote to replace the makefile:

#!/usr/bin/env python

# Build script for Sphinx documentation

import os
import shlex
import shutil
import subprocess
import sys

from collections import OrderedDict


# You can set these variables from the command line.
SPHINXOPTS = os.getenv('SPHINXOPTS', '')
SPHINXBUILD = os.getenv('SPHINXBUILD', 'sphinx-build')
PAPER = os.getenv('PAPER', None)
BUILDDIR = os.getenv('BUILDDIR', '_build')


TARGETS = OrderedDict()


def target(function):
    TARGETS[function.__name__] = function
    return function


# User-friendly check for sphinx-build
def check_sphinx_build():
    with open(os.devnull, 'w') as devnull:
        try:
            if subprocess.call([SPHINXBUILD, '--version'],
                               stdout=devnull, stderr=devnull) == 0:
                return
        except FileNotFoundError:
            pass
    print("The '{0}' command was not found. Make sure you have Sphinx "
          "installed, then set the SPHINXBUILD environment variable "
          "to point to the full path of the '{0}' executable. "
          "Alternatively you can add the directory with the "
          "executable to your PATH. If you don't have Sphinx "
          "installed, grab it from http://sphinx-doc.org/)"
          .format(SPHINXBUILD))
    sys.exit(1)


@target
def all():
    """the default target"""
    return html()


@target
def clean():
    """remove the build directory"""
    shutil.rmtree(BUILDDIR, ignore_errors=True)


def build(builder, success_msg=None, extra_opts=None, outdir=None,
          doctrees=True):
    builddir = os.path.join(BUILDDIR, outdir or builder)
    command = [SPHINXBUILD, '-b', builder]
    if doctrees:
        command.extend(['-d', os.path.join(BUILDDIR, 'doctrees')])
    if extra_opts:
        command.extend(extra_opts)
    command.extend(shlex.split(SPHINXOPTS))
    command.extend(['.', builddir])
    print(' '.join(command))
    if subprocess.call(command) == 0:
        print('Build finished. ' + success_msg.format(builddir))


@target
def html():
    """make standalone HTML files"""
    return build('html', 'The HTML pages are in {}.')


@target
def dirhtml():
    """make HTML files named index.html in directories"""
    return build('dirhtml', 'The HTML pages are in {}')


@target
def singlehtml():
    """make a single large HTML file"""
    return build('singlehtml', 'The HTML page is in {}.')


@target
def pickle():
    """make pickle files"""
    return build('pickle', 'Now you can process the pickle files.')


@target
def json():
    """make JSON files"""
    return build('json', 'Now you can process the JSON files.')


@target
def htmlhelp():
    """make HTML files and a HTML help project"""
    return build('htmlhelp', 'Now you can run HTML Help Workshop with the '
                             '.hhp project file in {}.')


@target
def qthelp():
    """make HTML files and a qthelp project"""
    return build('qthelp', 'Now you can run "qcollectiongenerator" with the '
                           '.qhcp project file in {0}, like this: \n'
                           '# qcollectiongenerator {0}/RinohType.qhcp\n'
                           'To view the help file:\n'
                           '# assistant -collectionFile {0}/RinohType.qhc')


@target
def devhelp():
    """make HTML files and a Devhelp project"""
    return build('devhelp', 'To view the help file:\n'
                            '# mkdir -p $HOME/.local/share/devhelp/RinohType\n'
                            '# ln -s {} $HOME/.local/share/devhelp/RinohType\n'
                            '# devhelp')


@target
def epub():
    """make an epub"""
    return build('epub', 'The epub file is in {}.')


@target
def rinoh():
    """make a PDF using rinohtype"""
    return build('rinoh', 'The PDF file is in {}.')


@target
def latex():
    """make LaTeX files, you can set PAPER=a4 or PAPER=letter"""
    extra_opts = ['-D', 'latex_paper_size={}'.format(PAPER)] if PAPER else None
    return build('latex', 'The LaTeX files are in {}.\n'
                          "Run 'make' in that directory to run these through "
                          "(pdf)latex (use the 'latexpdf' target to do that "
                          "automatically).", extra_opts)


@target
def latexpdf():
    """make LaTeX files and run them through pdflatex"""
    rc = latex()
    print('Running LaTeX files through pdflatex...')
    builddir = os.path.join(BUILDDIR, 'latex')
    subprocess.call(['make', '-C', builddir, 'all-pdf'])
    print('pdflatex finished; the PDF files are in {}.'.format(builddir))


@target
def latexpdfja():
    """make LaTeX files and run them through platex/dvipdfmx"""
    rc = latex()
    print('Running LaTeX files through platex and dvipdfmx...')
    builddir = os.path.join(BUILDDIR, 'latex')
    subprocess.call(['make', '-C', builddir, 'all-pdf-ja'])
    print('pdflatex finished; the PDF files are in {}.'.format(builddir))


@target
def text():
    """make text files"""
    return build('text', 'The text files are in {}.')


@target
def man():
    """make manual pages"""
    return build('man', 'The manual pages are in {}.')


@target
def texinfo():
    """make Texinfo files"""
    return build('texinfo', 'The Texinfo files are in {}.\n'
                            "Run 'make' in that directory to run these "
                            "through makeinfo (use the 'info' target to do "
                            "that automatically).")


@target
def info():
    """make Texinfo files and run them through makeinfo"""
    rc = texinfo()
    print('Running Texinfo files through makeinfo...')
    builddir = os.path.join(BUILDDIR, 'texinfo')
    subprocess.call(['make', '-C', builddir, 'info'])
    print('makeinfo finished; the Info files are in {}.'.format(builddir))


@target
def gettext():
    """make PO message catalogs"""
    return build('gettext', 'The message catalogs are in {}.', outdir='locale',
                 doctrees=False)


@target
def changes():
    """make an overview of all changed/added/deprecated items"""
    return build('changes', 'The overview file is in {}.')

@target
def xml():
    """make Docutils-native XML files"""
    return build('xml', 'The XML files are in {}.')


@target
def pseudoxml():
    """make pseudoxml-XML files for display purposes"""
    return build('pseudoxml', 'The pseudo-XML files are in {}.')


@target
def linkcheck():
    """check all external links for integrity"""
    return build('linkcheck', 'Look for any errors in the above output or in '
                              '{}/output.txt.')


@target
def doctest():
    """run all doctests embedded in the documentation (if enabled)"""
    return build('doctest', 'Look at the results in {}/output.txt.')


@target
def help():
    """List all targets"""
    print("Please use '{} <target>' where <target> is one of"
          .format(sys.argv[0]))
    width = max(len(name) for name in TARGETS)
    for name, target in TARGETS.items():
        print('  {name:{width}} {descr}'.format(name=name, width=width,
                                                descr=target.__doc__))


if __name__ == '__main__':
    check_sphinx_build()
    args = sys.argv[1:] or ['all']
    for arg in args:
        TARGETS[arg]()

Like the makefile, this still includes the project name (RinohType) here and there, which should be moved to a variable.

Some targets (latexpdf, info) also call make on a generated makefile. I think these should also be replaced with Python scripts.

[tk0miya]

The Makefile is well known and commonly used command. So to provide the Makefile helps the users who don't have python knowledges.
So I would like to keep providing Makefile for users.

BTW, better command line tools are welcome. Since 1.4, we provide make-mode (-M option) to sphinx-build. It tries to build document from python side. But I feel sphinx-build is still hard to use barely. So +1 for improving the command.

[shimizukawa]

+1 to keep providing Makefile/make.bat.
+1 for improving the sphinx-build command as tk0miya mentioned.

Just idea:

$ sphinx build html
$ sphinx quickstart
$ sphinx apidoc

[brechtm]

I didn't know about "make mode". This is even better than a Makefile-esque Python script!

I agree that there is room for improvement though:

• sphinx-build -h does not list the -M option
• the documentation only mentions make mode on the sphinx-quickstart page
• sphinx-build -M prints Error: at least 3 arguments (builder, source dir, build dir) are required.
• sphinx-build -M help prints the same, requiring the source and build directories to be specified as well, which makes no sense

And how can one add a "make target" for another builder? It would be nice if sphinx-build -M would accept and list (on -M help) all available (installed) builders). Each builder could supply the string to be displayed on -M help. Builders can be discovered using entry points as suggested in #2803. What do you think?

This wouldn't cover all make targets (such as latexpdf), which add another step after building with Sphinx. Would it make sense to add separate builders for this which extend the builder they depend on?

[brechtm]

Another thought: why not specify the source and build directories in conf.py? I think that would make sense. This would make them optional arguments to sphinx-build (still allowing to override).

[tk0miya]

Sorry, I don't know much about make-mode. But it looks not extensible with APIs.

Builders can be discovered using entry points as suggested in #2803. What do you think?

Personally, I agree that.
But, I heard the proposal using entry points was rejected once (before I joined to maintainers).
@shimizukawa Do you know about that?

Another thought: why not specify the source and build directories in conf.py? I think that would make sense. This would make them optional arguments to sphinx-build (still allowing to override).

If the environment is different, the paths are also different.
So I think it is not a configuration variable.

BTW, +1 for optional argument. It would be useful.

[brechtm]

If the environment is different, the paths are also different.
So I think it is not a configuration variable.

I'm not sure what you mean with "environment" here. In conf.py, I would specify the build and source directories passed to sphinx-build -M. The build directory is the "top-level" build directory (such as _build/), not the build directory specific to a particular builder.

[tk0miya]

The build directory is the "top-level" build directory (such as _build/), not the build directory specific to a particular builder.

The "top-level" build directory you said is not used in Sphinx application.
Sphinx application only uses an absolute build path to specific builder like /Users/tkomiya/work/tmp/doc/_build/html.
The path is different if users, host, OS and so on are different. This is "environment" I said.

Anyway, the Sphinx object requires both paths to instantiate. And conf.py are read after Sphinx app invoked.
So it is hard to specify the paths from conf.py during Sphinx 1.x series.

[brechtm]

@tk0miya Thanks for the explanation. Now I understand the problem.

But wouldn't it be possible for sphinx-build to also read conf.py, only extracting the values for the source and build directories? The Sphinx application can ignore them.

[tk0miya]

Yes, it's possible. But I can't still understand the advantage of conf.py.
Could you tell me the worth of your way?

[brechtm]

Currently they are specified in the Makefile, but I think the source and build directory paths belong in conf.py, just like other paths such as templates_path and html_static_path. In fact, the build path is currently duplicated in conf.py in the exclude_patterns variable.

If they are specified in conf.py, sphinx-build (in make mode) can retrieve them from there (but still allow overriding them). This would simplify sphinx-build -M invocation significantly:

sphinx-build -M html

Or, similar to @shimizukawa's suggestion above:

sphinx build html
# or perhaps
sphinx make html

[bskinn]

@tk0miya -- if the -M option is supposed to be part of the public command line API, it is not documented as such in the current stable documentation.