LilyPond - how to write a basic makefile

Question

During the last years I tought myself to write scores using LilyPond and furthermore to make file handling easier by using Bash scripts. Recently I started a project a little bigger than only one or two pages and once again I came across the article about Makefiles in the LilyPond Documentary.

Even though reading this article it was kinda hard for me to grasp the actual technique behind the template Makefile also because the article does not supply the file repository it is working with.

So I though to take the time to make up an example project to ask, how a Makefile in this scenario could look like. How I would build it step by step and how I actually run the Makefile. (Edit: referring to a make man page I got my understanding as far as being able to say that one uses make as interpreter like bash on a shell-script.sh. The command then simply looks like make -f Makefile run in the root directory of the project.)

---

The Project has a file-structure like this:

├── Book.ly
├── Book.pdf
├── global-files
│   ├── copyright.ily
│   ├── Frontpage.ily
│   ├── header.ily
│   └── paper.ily
├── input-files-voiceI
│   ├── Nr_01-voiceI.ily
│   ├── Nr_02-voiceI.ily
│   └── Nr_03-voiceI.ily
├── input-files-voiceII
│   ├── Nr_01-voiceII.ily
│   ├── Nr_02-voiceII.ily
│   └── Nr_03-voiceII.ily
├── README.md
├── single-pages-voiceI
│   ├── MIDI
│   │   ├── Score-Nr_01-voiceI.midi
│   │   ├── Score-Nr_02-voiceI.midi
│   │   └── Score-Nr_03-voiceI.midi
│   ├── PDF
│   │   ├── Score-Nr_01-voiceI.pdf
│   │   ├── Score-Nr_02-voiceI.pdf
│   │   └── Score-Nr_03-voiceI.pdf
│   ├── Score-Nr_01-voiceI.ly
│   ├── Score-Nr_02-voiceI.ly
│   └── Score-Nr_03-voiceI.ly
├── single-pages-voiceI_a_II
│   ├── MIDI
│   │   ├── Score-I_u_II_Nr_01.midi
│   │   ├── Score-I_u_II_Nr_02.midi
│   │   └── Score-I_u_II_Nr_03.midi
│   ├── PDF
│   │   ├── Score-I_u_II_Nr_01.pdf
│   │   ├── Score-I_u_II_Nr_02.pdf
│   │   └── Score-I_u_II_Nr_03.pdf
│   ├── Score-I_u_II_Nr_01.ly
│   ├── Score-I_u_II_Nr_02.ly
│   └── Score-I_u_II_Nr_03.ly
└── single-pages-voiceII
    ├── MIDI
    │   ├── Score-Nr_01-voiceII.midi
    │   ├── Score-Nr_02-voiceII.midi
    │   └── Score-Nr_03-voiceII.midi
    ├── PDF
    │   ├── Score-Nr_01-voiceII.pdf
    │   ├── Score-Nr_02-voiceII.pdf
    │   └── Score-Nr_03-voiceII.pdf
    ├── Score-Nr_01-voiceII.ly
    ├── Score-Nr_02-voiceII.ly
    └── Score-Nr_03-voiceII.ly

---

The input-files of both voices have a format like this:

\relative c {
  \clef bass
  \time 3/4
  \key c major

  c4( d e f      | %01
  g1) \bar "|."  | %02
}

---

The Score files have the purpose to be compiled for output of PDF and MIDI. and simply look like this (despite the fact that the Scores for two systems contain another Staff):

\version "2.18.2"

#(set-default-paper-size "a4")
#(set-global-staff-size 22)

\include "../global-files/header.ily"

\score {
  \new StaffGroup = "" \with {
    instrumentName = \markup { \bold \huge { \larger "1." }}
  }
  <<
    \new Staff = "celloI" \with { midiInstrument = #"cello" }

    \include "../input-files-voiceI/Nr_01-voiceI.ily"
  >>
  \layout {}
  \midi {}
}

---

The Book part is the part I'm still pretty unhappy with. I would prefer this pretty simple with just using the Score*.ly files as \includes, but I get problems with the \includes that are already in the Score.ly files, since they do not only contain the \score block to be compileable by them selfs.

Well I could use a \book with setting a book output name like \bookOutputSuffix "OutputName", but then my Book.ly would become a massive file, taking quite a long time to be compiled, even for a small change on a single piece.

So right now my Book.ly file has the following format and the only purpose to compile the whole book with two voices in two staffs, but with all pieces, here 01-03:

\version "2.18.2"

#(set-default-paper-size "a4")
#(set-global-staff-size 22)

\include "./global-files/paper.ily"

\book {

  \include "./global-files/Frontpage.ily"

  %%%% Score Number: 1 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "1." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_01-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_01-voiceII.ily"
    >>
    \layout {
      \printTupletBow
    }
  }

  %%%% Score Number: 2 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "2." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_02-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_02-voiceII.ily"
    >>
    \layout {}
  }

  %%%% Score Number: 3 ==================================%%%%

  \score {
    \new StaffGroup = "" \with {
      instrumentName = \markup { \bold \huge { \larger "3." }}}
    <<
      \new Staff = "voiceI" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceI//Nr_03-voiceI.ily"
      \new Staff = "voiceII" \with { midiInstrument = #"voice" }
      \include "./input-files-voiceII//Nr_03-voiceII.ily"
    >>
    \layout {}
  }
}

---

My workflow is the following:

1. I write the input files: input-file.ily
2. I run a bash-script.sh that creates the compileable Score.ly files from the input-files/*.ily
3. I run a bash-script.sh that creates the compileable Book.ly file from the input-files/*.ily
4. I compile the Score.ly files one by one or run a simple for file in *.ly; do lilypond "$file"; done loop, but in each of the three Score directories. I use a script to move the PDF and MIDI files into their corresponding folders.
5. I simply run lilypond to compile the Book.ly file.

DONE

---

The actual project this question is asked for can be found here on GitHub

---

Update 1:

My system:

    Operating System: Debian GNU/Linux bullseye/sid
              Kernel: Linux 5.3.0-2-686-pae
        Architecture: x86
        GNU LilyPond: 2.18.2
My Editor - GNU Nano: 4.5
      Guake Terminal: 3.6.3
            GNU Make: 4.2.1

I added my shell-scripts to a separate Git-Repository

---

Update 2:

This is a very simplified chart of the dependencies. Assuming there was only one voice:

./infiles/                        
  infile{01..03}.ily -------------> ./Book.ly ===> Book.pdf
     |                                 ^ ^ ^
     *---------> Scores{01..03}.ly === | | |=====> Score{01..03}.pdf
                      ^  ^         === | | |=====> Score{01..03}.midi
                      |  |             | | |
./global-files/       |  |             | | |
  header.ily    ------*  |             | | |
  copyright.ily ---------+-------------* | |
  Frontpage.ily -------------------------* |
  paper.ily     ---------------------------*

Answer 1

Invocation

make will search the current directory for a file named Makefile or makefile, so it's often simplest to name it one of these two choices and then invoke with the simple command:

$ make

If you use the uppercase 'M' then the file will usually be listed at the top according to alphabetical or collation order.

Rules

make operates by using rules for how to create an output from an input. The format for a rule is the target followed by a colon, then a space delimited list of dependencies, followed by commands indented with TAB.

target: dependencies
    commands

The target or dependencies can be filenames or symbols that correspond to other rules.

If you don't specify a target on the command line, it will invoke the first rule it encounters. So a common technique is to make the first rule a "dummy" rule which doesn't produce a file but simply collects all the steps or outputs together. Eg.

all: output1

Invoking make with this makefile will try to create output1 if it doesn't exist. If there's a rule for creating output1 later in the makefile it will use that.

For your case, I suggest making a top level rule to create Score.ly and Book.ly

all: Score.ly Book.ly

Pattern rules

To replace your shell loop, you can use a pattern rule.

%.pdf: %.ly
    lilypond $^

This rule says: To make a .pdf file, run lilypond on the corresponding .ly file.

Note that the command to execute must start with a literal TAB character. The %^ variable refers to the input file mentioned above. Other useful variables are $@ for the target, and $< for the first input if there are more than one.

This handles just part of your shell loop, defining the transformation from an input file to an output file. For the other task of generating a list of files, there are some special variables available in GNU make for this.

inputs= $(notdir $(wildcard ./*.ly))
bases= $(basename $(inputs))
outputs= $(patsubst %,%.pdf,$(bases))

After these definitions, you can use $(outputs) as a dependency in a rule, like:

pdfs: $(outputs)
    mkdir -p PDF
    mv *.pdf PDF
    mkdir -p MIDI
    mv *.midi MIDI

This example from the LilyPond docs shows that you can put multiple targets to the left of the colon, so you can account for both types of files produced by LilyPond which my example here doesn't do. Again, each of these commands must be indented with a bona fide ASCII TAB character.

For very simple cases

You can use a makefile just to collect one or more shell scripts together, ignoring much of the complexity of rules and dependencies if your situation is very simple.

For one project where I made a bunch of .abc files in a single directory, the whole thing is handled by a single rule:

all:
    for f in `ls *.abc` ;               \
    do ../abcm2ps -O $${f%.abc}.ps $$f ; \
       ps2pdf $${f%.abc}.ps ;           \
    done ;
    zip -r evildead.zip *.pdf

Answer 2

luser droog’s answer gives a good overview of make itself. Here’s an example of how to apply that to a real-world Lilypond project with the following characteristics:

• Multiple parts: piano, bassoon, etc.
• Multiple movements.
• Multiple desired outputs: a single “master” (conductor) score, a master score for each movement, and a part score for each instrument.
• Both PDF and MIDI outputs.
• Common definitions (macros) for both music and presentation.

For instance, if your Book.ly file includes your Nr_01-voiceI.ily file, which in turn includes your macros.ily shared definitions file, then if macros.ily is changed, your Makefile needs to know that it needs to recompile Book.ly to update Book.pdf.

On the othre hand, if you change just one part, it is very convenient to be able to write make parts and have Make only recompile the part that was changed, rather than wasting time recompiling all the others. And, if you just want to listen to a segment, you should be able to make midi to skip the slow typesetting and only re-do the necessary sequencing.

I’ve written a shell script to generate a Makefile that has all these properties. In particular, it automatically traverses the \include graph of your project to figure out which Lilypond files depend on which others, and will incur the minimal set of recompilations for any particular change. You just tell it what “main files” you have, and which ones you want in PDF and/or MIDI form, and it will do the rest.

Here’s the script: https://github.com/MutopiaProject/MutopiaProject/blob/918971593735f2dbf4864f289767b8d59a7d950e/ftp/MozartWA/KV488/Mozart-KV488/Mozart-KV488-lys/create_makefile.sh

Here’s an example of the output Makefile: https://github.com/MutopiaProject/MutopiaProject/blob/918971593735f2dbf4864f289767b8d59a7d950e/ftp/MozartWA/KV488/Mozart-KV488/Mozart-KV488-lys/Makefile

The core mechanism is to create a secondary target for each Lilypond file, which I’ve called its “lydep” (“Lilypond dependencies”) file, such that a file’s lydep is dirty if and only if any of the file’s transitive sources is dirty. In practical terms, this means that each lydep target needs to depend on its source file plus all the lydep targets on which it has direct \include dependencies. Then, Make’s automatic resolution takes care of the rest.

The script is tailored to a particular project, but you should be able to keep the core infrastructure in place and customize the makefile function at the bottom to substitute in your project structure.

I’ve written this script for GNU/Linux, and it might need some minor adjustments for macOS/BSD (e.g., swapping out readlink -f), but it’s pretty straightforward overall. It does require bash and GNU make (for secondary targets). I see that you’re on Debian, so it should run just fine as is.

This is all released under the MIT License. Please feel free to take whatever you find helpful.