#Table of Contents

1. About
   1. Why Does This Exist?
2. Parallelism
3. Content Hash Based Builds
4. Terminology
5. Projects
   1. Ocaml
   2. Third-party
6. Dependencies
7. Configuration
   1. Configuration Variables For Projects
   2. Configuration Variables For Tests
   3. Outputting the configuration
8. Produced Files
9. Documentation
10. Selectors
11. Changelog
    1. 1.0.0
    2. 1.1.0
    3. 2.0.0
    4. 3.0.0
    5. 3.0.1
    6. 3.0.2
    7. 3.0.3
    8. 3.0.4
    9. 3.1.0
    10. 3.1.1
    11. 4.11
    12. 4.12
    13. 4.13
    14. 5.14
    15. 5.15
    16. 5.16
    17. 5.17 (bug release)
    18. 5.18
    19. 5.19
    20. 6.20
    21. 5.21
    22. 5.22
    23. 5.23
    24. 5.24
    25. 5.25
    26. 5.26 (bug release)
    27. 5.27
    28. 5.28
    29. 5.29
    30. 6.43
    31. 6.44
    32. 6.45
    33. 6.46
    34. 6.47
    35. 6.48
    36. 6.49
    37. 6.50

#About

pds is a build system for Ocaml that is meant to make it easy to build a project that follows a particular layout by generating a makefile for the project. The input to pds is a config file, pds.conf, and a directory structure, which is always the current working directory, and the output is the build description.

The directory layout looks like:

.
 /src
     /proj1
     /proj2
     /proj3
 /tests
    /test1/test.ml
    /test2/test.ml

The pds.conf for this project might look like:

[src.proj1]
type = "exec"
deps = [ "core", "proj2", "proj3" ]
install = true
install_cmd = "cp -vf proj1.native $(PREFIX)/bin/proj1"
remove_cmd = "rm -v $(PREFIX)/bin/proj1"

[src.proj2]
deps = [ "proj3" ]
install = false

[src.proj3]
deps = [ "str" ]
install = false

[tests.test1]
deps = [ "proj2" ]

[tests.test2]
deps = [ "proj1" ]

Finally, by using the standard Makefile definition below, make test will compile all the code and run the tests. make install will install proj1. And make remove will remove it.

all:
	pds
	$(MAKE) -f pds.mk all

%:
	pds
	$(MAKE) -f pds.mk $*

pds supports multiple types of builds, at this time release, debug, profile, and docs. The release build must always exist and is what running make with no target will execute. The debug and profile builds will inherit all configuration from the release build and options can be overridden. pds also supports tests, and the existing test targets are: test, test-debug, and test-profile, which run tests for the respective builds.

pds builds into the build directory and each build type is a sub-directory in the build directory, for example build/release, build/docs, or build/test-release.

#Why Does This Exist?

Ocaml has multiple build systems, so it is probably confusing and frustrating to learn that another one has been created. In the author's experience, many of the Ocaml build tools are more complicated than the value they add. pds is meant to be simple but play nice with other tools. If you've felt frustrated by the existing tooling, then pds might work well for you. If you are happy with the existing tooling then you should be able to safely ignore pds because it meant to be non-intrusive.

#Parallelism

pds supports parallel builds. The amount of parallelism is specified with the -j option in make. A repository with multiple projects in it will parallelize the build across those projects. That is, if there is proj1 and proj2 that do not depend on each other, they will be built in parallel. Inside of an Ocaml project, if every .ml file in the project has an associated .mli file, the project will be built in parallel. If there is any .ml file that does not have a .mli file, the project will be built in serial. The reason for this is when compiling a .ml file that does not have a .mli file, both ocamlc and ocamlopt generate a .cmi file even if the .cmi file already exists. With a parallel build, the byte code and native code builds will be executed in parallel. If the project is built in parallel, the generation of the .cmi file can overlap, generating compiler errors. This is the source of the units Foo and Bar make inconsistent assumptions about Baz error.

#Content Hash Based Builds

pds does some trickery to build changes only if the contents have changed and not based on timestamp.

To accomplish this, pds stores all input files and a hash in a sqlite database and consults this on each run. If the hash of a file is the same but its timestamp has changed, pds sets the timestamp to the value in the database. This tricks Make into only building things that have actually changed.

If the hashes do not match, pds sets the timestamp to that of the current time and stores that value in the database.

The hash stored for every file is both the hash of the file as well as the hash of the build configuration, this ensure that if a pds configuration has changed that causes a rebuild.

#Terminology

• Project - The name of a directory in the src directory. This corresponds to an entry in the src table in the pds configuration file.
• Build - A kind of build, for example release, debug, profile or docs.
• Project type - The type of project it is, either ocaml or third-party.
• Project target type - The type of output the project has, either exec or library, this applies only to ocaml projects..

#Projects

Projects are divided up between Ocaml projects, which pds knows how to generate a Makefile for and a third-party project, which pds does not generate a Makefile for but are expected to have one that corresponds to the pds interface, which is release, debug, profile, install, remove, docs, and clean.

#Ocaml

Ocaml projects can be either executables or libraries. The name of the output is derived from the project name with a suffix added depending on the type. In the case of a library, a META file compatible with ocamlfind will be generated.

Project Type	Suffix
Library	cma
Library	cmxa
Executable	byte
Executable	native

#Third-party

Third party projects do not have a Makefile generated by pds but are expected to have one which corresponds to the pds interface.

#Dependencies

The configuration file input to pds specifies dependencies, if any, for each project. If a dependency has the same name as a project in the src directory then it is considered an internal dependency and an internal dependency will be compiled before its dependent project and the emitted output will be specified as a dependency in the dependents Makefile.

#Configuration

A configuration file is required for all projects. A configuration file path can be given explicitly or pds will search for pds.conf in the current working directory. pds.conf is a toml file. All directories in the src directory must be in the configuration file. pds will error if that is not the case. The only required attribute for each project is install, which specifies if it will be installed or not. In the case of an exec project, the install_cmd and remove_cmd are required.

Projects and tests can have some configuration values overridden by specific build types, the options and precedence is given in the respective sections below.

An example of building the example directory structure.

• proj3 requires no special dependencies or settings and projects are assumed to be library.

• The global release build specifies a project should be compiled with -noassert.

• The global debug build specifies that a project should be compiled with -g.

• The project proj2 overrides the global compiler options for release and debug and compiles with -safe-string.

• The debug build options completely override the release build options.

• Libraries have a default ocamlfind install directory, however, commands need to have their installation command specified.

  [global.release] extra_compiler_opts = "-noassert"

  [global.debug] extra_compiler_opts = "-g"

  [src.proj1] deps = ["async", "proj2", "proj3"] type = "exec" install = true install_cmd = "cp proj1.native $(PREFIX)/bin/proj1" remove_cmd = "rm $(PREFIX)/bin/proj1"

  [src.proj2] install = false deps = ["core", "async"] type = "library" extra_compiler_opts = "-safe-string -noassert" debug = { extra_compiler_opts = "-safe-string -g" }

  [src.proj3] install = false

#Configuration Variables For Projects

Name	Type	Default	Description
deps	string list	[]	List of dependencies both within the project and external
type	string	"library"	exec, library - Whether the ocaml project is an executbale or library
install	bool		Whether the built artifacts should be installed
installcmd	string		If the type is exec, specify the command to be used to install it
removecmd	string		If the typs is exec, specify the command to be used to remove it
projecttype	string	"ocaml"	third-party or ocaml - defaults to ocaml
extracompileropts	string	""	Extra options to give to the ocaml compiler
extraocamldepopts	string	""	Extra options to give ot ocamldep
extramakefilelines	string list	[]	Extra lines to add to the generated Makefile
compiledeps	string list	[]	List of builds that need to be built prior to this build but are not automatically linked against
metalinkopts	string	""	Link options to go into the META file.
build	bool	true	Whether or not to build the project or test

The following values can be overridden:

• extra_compiler_opts
• extra_ocamldep_opts
• deps - Note that this has no entry in global.
• compile_deps - Note that this one has no entry in global.

The precedence ordering is given below. Selectors are discussed later in the document.

For release builds:

1. src.<project>.selector.<selector>.<option>
2. src.<project>.<option>
3. global.release.<option>

For all other builds:

1. src.<project>.selector.<selector>.<build_type>.<option>
2. src.<project>.selector.<selector>.<option>
3. src.<project>.<build_type>.<option>
4. global.<build_type>.<option>
5. src.<project>.<option>

The same precedence applies to test builds.

#Configuration Variables For Tests

Tests only support specifying the deps and extra_compiler_opts configuration. The deps option cannot be specified in the global section.

The precedence ordering is given below. Note that release builds are unique in that they search the third precedence first then the second.

1. tests.<project>.<build_type>.<option>
2. global.test-<build_type>.<option>
3. tests.<project>.<option>

#Outputting the configuration

This feature is not experimental and will likely change.

pds can output the configuration in a tab delimited format, like a CSV. The output looks like the following:

Column	Description
Source Type	"src" or "test", the type of build it is
Name	The name of the build, "src.<name>" or "test.<name>"
Build Type	"exec" or "library", empty for test
Project Type	"ocaml" or "third-party", empty for a test
Deps	Comma separated list of dependencies

#Produced Files

pds will produce files when it is executed as well as when the build is performed. This is a list of files that will be produced and can be safely ignored in the form of a .gitignore.

build/
pds.mk
Ocamlrules.mk.in

#Documentation

pds can generate HTML documentation using ocamldoc. The docs make target is automatically generated and third-party libraries are expected to implement a docs target. The docs are put into the builds/docs/<project> of the root directory of the repository. By default, only mli files are included in documentation, however this can be modified by setting the DOCS_FILES variable using the extra_makefile_lines configuration setting. Different options can also be set for ocamldoc by modifying the OCAMLDOC_OPTS variable through extra_makefile_lines.

#Selectors

Some builds need to be parameterized over the environment it's being compiled in. This is accomplished through selectors. A selector is a program which outputs a single line that matches the regex [a-zA-Z0-9_-]+, any trailing white space will be removed. The value of the line will then be used to look up values in src.<project>.selector.<line>.<variable>. For any selection not in src.<project>.selector.<line> the value at the top-level of a project definition is used. The empty string is a special selector which utilizes the default project values.

The only override supported by selectors is setting the variable. It is not possible to unset a variable in a selector.

Take the case where a project should use different dependencies if it is executed in environment A vs environment B. The selector program, is_a_or_b will return the line A or B depending on the system.

[global]
selector = ["is_a_or_b"]

[src.foo]
deps = ["dep1", "dep2"]

[src.foo.selector.A]
deps = ["dep1", "dep2", "dep3"]

[src.foo.selector.B]
deps = ["dep1, "dep2", "dep4"]

[src.bar]
deps = ["depX", "depY" ]

[src.bar.selector.A]
deps = ["dep1", "dep2", "dep3"]

Any variable that applies to a project can be set in a selector.

The order of precedence is that the selector for the build type is first checked, then the selector for a release.

#Changelog

#1.0.0

• Build makefiles of Ocaml projects.
• Add support for third-party projects.
• Support running tests.

#1.1.0

• Add support for compile deps. These are internal dependencies that are not an ocamlfind dependency but need to be build before building the project. This is useful if a tool needs to be built that will be used to compile a project.

#2.0.0

• Remove globals section. This can become confusing when for some configuration variables as it's unclear where their values come from.
• Fill out the documentation more.

#3.0.0

• Add remove support. This was previously outside of pds but now pds itself understands both installing and uninstalling projects.

#3.0.1

• Remove pds from the package dependencies, this was previously done by hand after using hll to make the package.

#3.0.2

• Expand the hll config to pass opam-linter.

#3.0.3

• Add crunch dependency to hll.
• Specify which version of ocaml pds works with.

#3.0.4

• Add more documentation.
• Remove examples.

#3.1.0

• Add initial ocamldocs support.
• Install .cmi files.

#3.1.1

• Fix a bug in generating docs when the project has no dependencies.

#4.11

• Switch to monotonic versioning.
• Add support for formatted output, a simple tab separated representation of a pds.conf.
• Rename the docs target to doc because this conflicts with the directory name of the output.

#4.12

• Include tests directories in formatted output.

#4.13

• Make a better error message for the install key, which is required.

#5.14

• Rework builds to use the build output directory rather than building in the same directory as the source files. This allows multiple build types to coexist, such as release and debug.
• Remove PACK support. Building PACKs is no longer supported.

#5.15

• Fix formatted output for third-party projects.
• Add a changelog and back fill it.

#5.16

• Fix bug in cleanup logic which concatenated multiple deps

#5.17 (bug release)

• Fix a build race condition during parallel builds. When a .cmi is generated from a .ml, in a parallel build it is possible that the build of the .cmo and .cmx will execute in parallel and both will generate a .cmi which will step on each other. Now, for files that will generate a .cmi, the .cmi is generated first then the .cmo and .cmx are generated.

#5.18

• 5.17 was a bug release, its solution made builds that could not be installed.
• This solves the problem 5.17 intended to solve by making ocaml builds that have any .ml files without a complimentary .mli file now build in serial. This is because building the byte-code for .ml files with no .mli file can overlap with each other generating the .cmi file which creates a race condition where the .cmi is in an invalid state when a later step is using it. This solves it by serializing the build for those cases.

#5.19

• Fix bug where it was only installing .cmi files for .mli files. Not it installs .cmi files for all .ml files.

#6.20

• Removed the -custom option from default byte code builds.

#5.21

• Undo the change in 6.20. Instead, split out the linker options and allow them to be overridden.

#5.22

• Support extra_makefile_lines in tests.
• Tests compile with OCAML*_LINK_OPTS just like regular builds.

#5.23

• Fix typo in PARALLEL

#5.24

• Build cmti files and install them for every library.

#5.25

• Support adding linkopts to the META file.

#5.26 (bug release)

• Support selectors.
• Add some tests.

#5.27

• Fix bug in section name for the selector.

#5.28

• Fix lookup order. This is a long-standing bug where selectors and globals were not properly looked up.

#5.29

• Support rebuilding when pds.conf changes.
• Remove support for specifying pds.conf (backwards breaking but it never worked).

#6.43

• Move to content based hashing.

#6.44

• Fix install directive

#6.45

• Better sqlite error reporting

#6.46

• strftime usage that works on Alpine

#6.47

• Another instance of strftime

#6.48

• Fix bugs in testing outputs.

#6.49

• Improve performance 10x by batching database operations.

#6.50

• Fix handling file renames or deletes. Pds would remake the file, thinking that its hash had been changed, when in reality it didn't exist.