Afraid of Makefiles? Don't be
I think the primary thing that makes people fear Makefiles is that they try learning it by inspecting the output of automake/autoconf, cmake, or other such systems. These machine-generated Makefiles are almost always awful to look at, primarily because they have several dozen workarounds and least-common-denominators for make implementations dating back to the 1980s.
A properly hand-tailored Makefile is a thing of beauty, and it is not difficult.
Make is awesome. I have always loved make, and got really good with some of its magic. After switching to Java years ago, we collectively decided, "platform independent tools are better", and then we used ant. Man was ant bad, but hey! It was platform independent.
Then we started using maven, and man, maven is ridiculously complex, especially adding custom tasks, but at least it was declarative. After getting into Rust, I have to say, Cargo got the declarative build just right.
But then, for some basic scripts I decided to pick Make back up. And I wondered, why did we move away from this? It's so simple and straightforward. My suggestion, like others are saying, is keep it simple. Try and make declarative files, without needing to customize to projects.
I do wish Make had a platform independent strict mode, because this is still an issue if you want to support different Unixes and Windows.
p.s. I just thought of an interesting project. Something like oh-my-zsh for common configs.
By using pseudo targets only in the example and not real files, the article misses the main point of targets and dependencies: target rules will only be executed if the dependencies changed. make will compare the time of last modification (mtime) on the filesystem to avoid unnecessary compilation. To me, this is the most important advantage of a proper Makefile over a simple shell script always executing lots of commands.
Sneaky pro-tip - use Makefiles to parallelize jobs that have nothing to do with building software. Then throw a -j16 or something at it and watch the magic happen.
I was stuck on an old DoD redhat box and it didn't have gnu parallel or other such things and co-worker suggested make. It was available and it did the job nicely.
Today's simple makefiles are the end result of lessons hard learned. You'd be horrified to see what the output of imake looked like.
From memory here's a Makefile that serves most of my needs (use tabs):
SOURCE=$(wildcard *.c)
OBJS=$(patsubst %.c,%.o, $(SOURCE))
CFLAGS=-Wall
# define CFLAGS and LDFLAGS as necessary
all: name_of_bin
name_of_bin: $(OBJS)
$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)
%.o: %.c
$(CC) $(CFLAGS) -o $@ $^
clean:
rm -f *.o name_of_bin
.PHONY: clean allDue to its versatility, Makefiles can be creatively used beyond building software projects. Case in point: I used a very simple hand-crafted Makefile [1] to drive massive Ansible deployment jobs (thousands of independently deployed hosts) and work around several Ansible design deficiencies (inability to run whole playbooks in parallel - not just individual tasks, hangs when deploying to hosts over unstable connection, etc.)
The principle was to create a make target and rule for every host. The rule runs ansible-playbook for this single host only. Running the playbook for e.g. 4 hosts in parallel was as simple as running 'make -j4'. At the end of the make rule, an empty file with the name of the host was created in the current directory - this file was the target of the rule - it prevented running Ansible for the same host again - kind of like Ansible retry file, only better.
I realize that Ansible probably is not the best tool for this kind of job, but this Makefile approach worked very well and was hacked together very quickly.
[1] https://gist.github.com/martinky/819ca4a9678dad554807b68705b...
I love Make for my small projects. It still could be better. Here is my list:
* Colorize errors
* Hide output unless the command fails
* Automatic help command which shows (non-file) targets
* Automatic clean command which deletes all intermediate files
* Hash-based update detection instead of mtime
* Changes in "Makefile" trigger rebuilds
* Parallel builds by default
* Handling multi-file outputs
* Continuous mode which watches the file system for changes and rebuilds automatically
I know of no build system which provides these features and is still simple and generic. Tup is close, but it fails with LaTeX, because of the circular dependencies (generates and reads aux file).
"Build systems are the bastard stepchild of every software project" -- me a years ago
I've work in embedded software for over a decade, and all projects have used Make.
I have a love-hate relationship with Make. It's powerful and effective at what it does, but its syntax is bad and it lacks good datastructures and some basic functions that are useful when your project reaches several hundred files and multiple outputs. In other words, it does not scale well.
Worth noting that JGC's Gnu Make Standard Library (GMSL) [1] appears to be a solution for some of that, though I haven't applied it to our current project yet.
Everyone ends up adding their own half-broken hacks to work around some of Make's limitations. Most commonly, extracting header file dependency from C files and integrating that into Make's dependency tree.
I've looked at alternative build systems. For blank-slate candidates, tup [2] seemed like the most interesting for doing native dependency extraction and leveraging Lua for its datastructures and functions (though I initially rejected it due the the silliness of its front page.) djb's redo [3] (implemented by apenwarr [4]) looked like another interesting concept, until you realize that punting on Make's macro syntax to the shell means the tool is only doing half the job: having a good language to specify your targets and dependency is actually most of the problem.
Oh, and while I'm around I'll reiterate my biggest gripe with Make: it has two mechanisms to keep "intermediate" files, .INTERMEDIATE and .PRECIOUS. The first does not take wildcard arguments, the second does but it also keeps any half-generated broken artifact if the build is interrupted, which is a great way to break your build. Please can someone better than me add wildcard support to .INTERMEDIATE.
[1] http://gmsl.sourceforge.net
[2] http://gittup.org/tup/ Also its creator, Mike Shal, now works at Mozilla on their build system
Makefiles are easy for small to medium sized projects with few configurations. After that it seems like people throw up their hands and use autotools to deal with all the recursive make file business.
Most attempts to improve build tools completely replace make rather than adding features. I like the basic simplicity and the syntax, (the tab thing is a bit annoying but easy enough to adapt to).
It'd be interesting to hear everyone's go to build tools.
> You've learned 90% of what you need to know about make.
That's probably in the ballpark, anyways.
The good (and horrible) stuff:
- implicit rules
- target specific variables
- functions
- includes
I find that with implicit rules and includes I can make really sane, 20-25 line makefiles that are not a nightmare to comprehend.
For a serious project of any scope, it's rare to use bare makefiles, though. recursive make, autotools/m4, cmake, etc all rear their beautiful/ugly heads soon enough.
But make is my go-to for a simple example/reproducible/portable test case.
I feel like any discussion of make is incomplete without a link to Recursive Make Considered Harmful[0]. Whether you agree with the premise or not, it does a nice job of introducing some advanced constructs that make supports and provides a non-contrived context in which you might use them.
The trouble with "make" is that it's supposed to be driven by dependencies, but in practice it's used as a scripting language. If the dependency stuff worked, you would never need
make clean; make
touchAlmost every build system (where I think it isn't controversial to say make is most often used) looks nice and simple with short, single-output examples to demonstrate the basis of a system.
It's when you start having hundreds of sources, targets, external dependencies, flags and special cases that it becomes hard to write sane, understandable Makefiles, which it presumably why people tend to use other systems to generate makefiles.
So sure, understanding what make is, and how it works is probably important, since it'll be around forever. But there are usually N better ways of expressing a build system, nowadays.
So I saw this and thought why not give it a try. How hard could it be right? My goal? Take my bash file that does just this (I started go just yesterday so I might be doing cross compiling wrong :D) :
```
export GOPATH=$(pwd)
export PATH=$PATH:$GOPATH/bin
go install target/to/build
export GOOS=darwin
export GOARCH=amd64
go install target/to/build
```
which should be simple. Right? Set environment variables, run a command. Set another environment variable, run a command.
45 minutes in and I haven't been able to quite figure it out just yet. I definitely figured out how to write my build.sh files in less than 15 minutes for sure when I started out.
One important tip is that the commands under a target each run sequentially, but in separate shells. So if you went to set env vars, cd, activate a Python virtualenv, etc to affect the next command, you need to make them a single command, like:
target:
cd ./dir; ./script.shThose who don't understand Make are condemned to reimplement it, poorly.
I remember trying to wrap my head around the monstrosity that is Webpack. Gave up and used make, never looked back since
Personal blog spam, I learned make recently too and discovered it was good for high level languages as well, here is an example of building a c# project: http://flukus.github.io/rediscovering-make.html .
Now the blog itself is built with make: http://flukus.github.io/building-a-blog-engine.html
If you want all the greatness of Makefiles without the painful syntax I can highly recommend Snakemake: https://snakemake.readthedocs.io/en/stable/
It has completely replaced Makefiles for me. It can be used to run shell commands just like make, but the fact that it is written in Python allows you to also run arbitrary Python code straight from the Makefile (Snakefile). So now instead of writing a command-line interface for each of my Python scripts, I can simply import the script in the Snakefile and call a function directly.
Eg.
rule make_plot:
input: data = "{name}.txt"
output: plot = "{name}.png"
run:
import my_package
my_package.plot(input['data'], output['plot'], name = wildcards['name'])
These days, most of my projects have a Makefile with four or five simple commands that _just work_ regardless of the language, runtime or operating system in use:
- make deps to setup/update dependencies
- make serve to start a local server
- make test to run automated tests
- make deploy to package/push to production
- make clean to remove previously built containers/binaries/whatever
There are usually a bunch of other more specific commands or targets (like dynamically defined targets to, say, scale-frontends-5 and other trickery), but this way I can switch to any project and get it running without bothering to lookup the npm/lein/Python incantation du jour.
Having sane, overridable (?=) defaults for environment variables is also great, and makes it very easy to do stuff like FOOBAR=/opt/scratch make serve for one-offs.
Dependency management is a much deeper and broader topic, but the usefulness of Makefiles to act as a living document of how to actually run your stuff (including documenting environment settings and build steps) shouldn't be ignored.
(Edit: mention defaults)
For people who are more comfortable in Python, I highly recommend Snakemake[1]. I use it for both big stuff like automating data analysis workflows and small stuff like building my Resume PDF from LyX source.
Make is fine for simple cases, but I'm working on a project that is based on buildroot right now, and it is kind of a nightmare: make just does not provide any good way at this scale to keep track of what's going on and inspect / understand what goes wrong. Especially in the context of a highly parallel build with some dependencies are gonna get missing.
In general also all the implicit it has makes it hard to predict what can happen. Again when you scale to support a project that would be 1) large and 2) wouldn't have a regular structure.
On another smaller scale: doing an incremental build of LLVM is a lot faster with Ninja compared to Make (crake-generated).
Make is great: just don't use it where it is not the best fit.
Here's some tips I like to follow whenever writing Makefiles (I find them joyful to write): http://clarkgrubb.com/makefile-style-guide
Please, don't ship your own Makefiles. Yes, autotools sucks - but there is one thing that sucks more: no "make uninstall" target.
Good people do not ship software without a way to get rid of it, if needed.
wow i've been feeling like not knowing make has been a major weakness of mine, this article has finally tied all my learning together. i feel totally capable of using make now. thank you.
Has anybody successfully used make to build java code? I realize there are any number of other options (ant, maven, and gradle arguably being the most popular).
In fact, I realize that the whole idea of using make is probably outright foolish owing to the intertwined nature of the classpath (which expresses runtime dependencies) and compile-time dependencies (which may not be available in compiled form on the classpath) in Java. I'm merely curious if it can be done.
This is great, and needs saying.
Recently I wrote a similar blog about an alternative app pattern that uses makefiles:
https://zwischenzugs.wordpress.com/2017/08/07/a-non-cloud-se...
Makefiles are simple, but 99% of the existing Makefiles are computer-generated incomprehensible blobs. I don't want that.
I did not know people are afraid of Makefiles. Maybe a naïve question, but what is so scary about make?
>>> Congratulations! You've learned 90% of what you need to know about
The next 90% will be to learn that Make breaks when having tabs and spaces in the same file, and your developers all use slightly different editors that will mix them up all the time.
Instead of makefile I can recommend Taskfile https://hackernoon.com/introducing-the-taskfile-5ddfe7ed83bd
Simple to use without any magic.
I had written Non Recursive Makefile Boilerplate (nrmb) for C, which should work in large projects with recursive directory structure. There is no need to manually add source file names in makefile, it automaically do this. One makefile compiles it all. Of course, it isn't perfect but it does the job and you can modify it for your project. Here is the link
https://github.com/quantos-unios/nrmb
Have a look :)
Make is fine, but I think we have better tools nowadays to do the same things.
Even though it may not have been originally intended as such, I've found Fabric http://docs.fabfile.org/en/1.13/tutorial.html to be far far more powerful and intuitive as a means of creating CLI's (that you can easily parametrize and test) around common tasks such as building software.
Or just use cmake and save yourself time, effort, and pain.
After using the various javascript build processes, I went back to good old makefiles and the result is way simpler. I have a target to build the final project with optimizations and a target to build a live-reload version of the project, that watches for changes on disk and rebuilds the parts as needed (thanks to watchify).
This works in my cases because I have browserify doing all the heavy lifting with respect to dependency management.
Opinion poll. I'm writing a little automation language in YAML and I was wondering if people prefer a dependency graph concept where tasks run parallel by default, unless stated as dependency, or a sequential set of instructions where tasks only run in parallel if explicitly "forked".
I'd say people would lean towards the former, but time and real world experience has shown that sequential dominates everything else.
I almost always roll a basic Makefile for even simple web projects. PHONY commands like "make run" and "make test" in every project make context switching a bit more easier.
While things like "npm start" are nice, not all projects are Node.js. In my current startup we're gonna have standardised Makefiles in each project so its easy to build, test, run, install any microservice locally :)
Using Cmake is so much nicer than make, and it's deeply cross-platform. Cmake makes cross-compiling really easy, while with make you have to be careful and preserve flags correctly. Much nicer to just include a cmake module that sets up everything for you. Plus it can generate xcode and visual studio configs for you. Doing make by hand just seems unncessary.
Other resources:
I haven't ever had to mess with Makefiles, as I don't program anything other the basic shell scripts, but I do remember reading articles in the past that indicated Makefiles can be used for more than just programming.
For example: using Makefiles to automate static webpage creation and image file conversion.
Take a look at the LibreOffice gbuild system, completely written in GNU make "language". And then come back saying you're not afraid of make ;-)
Still, it probably would be much harder, if possible at all (doubted for most), to achieve the same with any other tool mentioned here.
Worth pointing to Mike Bostock's essay, Why Use Make: https://bost.ocks.org/mike/make/
Who needs makefiles when you have a build system, you might ask.
The truth: go see the configuration of a Jenkins project, and there's a high chance that one of the lines there still says "make".
> Add an @ sign to suppress output of the command that is executed.
This is the exact opposite. It supresses echoing the command that is being executed. It's output is still shown like normal.
Delphi/Pascal haven't used make files for...I don't know how long.
Shout out to FreePascal/Lazarus yet again!
So what is npm install of C/C++ ?
so I guess makefile is somewhat like gulp JS... but can I split a makefile into multiple files?
I'm not scared of Makefiles, I just find them painful to work with.
I like Tup
One very important thing missing from this primer is that Make targets are not 'mini-scripts', even though they look like it. Every line is 'its own script' in its own subshell - state is not passed between lines.
Make is scary because it's arcane and contains a lot of gotcha rules. I avoided learning Make for a long time. I'm glad I did learn it in the end, though I wouldn't call myself properly fluent in it yet. But there are a ton of gotchas and historical artifacts in Make.
I just always hated that you're supposed to write ".PHONY". The first time I forgot to do that and it built a random file named after a build step I'd scrap the whole thing.
Is that a rhetorical question? Is there anyone who is afraid of makefiles? Makefiles exist to make your life easier.
Make's underlying design is great (it builds a DAG of dependencies, which allows for parallel walking of the graph), but there's a number of practical problems that make it a royal pain to use as a generic build system:
1. Using make in a CI system doesn't really work, because of the way it handles conditional building based on mtime. Sometimes you just don't want the condition to be based on mtime, but rather a deterministic hash, or something else entirely.
2. Make is _really_ hard to use to try to compose a large build system from small re-usable steps. If you try to break it up into multiple Makefiles, you lose all of the benefits of a single connected graph. Read the article about why recursive make is harmful: http://aegis.sourceforge.net/auug97.pdf
3. Let's be honest, nobody really wants to learn Makefile syntax.
As a shameless plug, I built a tool similar to Make and redo, but just allows you to describe everything as a set of executables. It still builds a DAG of the dependencies, and allows you to compose massive build systems from smaller components: https://github.com/ejholmes/walk. You can use this to build anything your heart desires, as long as you can describe it as a graph of dependencies.