Building Plugins

Rebar3’s system is based on the concept of providers. A provider has three callbacks:

  • init(State) -> {ok, NewState}, which helps set up the state required, state dependencies, etc.
  • do(State) -> {ok, NewState} | {error, Error}, which does the actual work.
  • format_error(Error) -> String, which allows to print errors when they happen, and to filter out sensitive elements from the state.

A provider should also be an OTP Library application, which can be fetched as any other Erlang dependency, except for Rebar3 rather than your own system or application.

This document contains the following elements:

Using a Plugin

To use the a plugin, add it to the rebar.config:

  1. {plugins, [
  2. {plugin_name, {git, "git@host:user/name-of-plugin.git", {tag, "1.0.0"}}}
  3. ]}.

Then you can just call it directly:

  1. $ rebar3 plugin_name
  2. ===> Fetching plugin_name
  3. ===> Compiling plugin_name
  4. <PLUGIN OUTPUT>

Reference

Provider Interface

Each provider has the following options available:

  • name: The ‘user friendly’ name of the task.
  • module: The module implementation of the task.
  • hooks: A two-tuple of provider names for pre and post-hooks ({Pre, Post}).
  • bare: Indicates whether task can be run by users or not. Should be true.
  • deps: The list of dependencies, providers that need to run before this one. You do not need to include the dependencies of your dependencies.
  • desc: The description for the task, used by rebar3 help
  • short_desc: A one line short description of the task, used in lists of providers
  • example: An example of the task usage, such as "rebar3 my-provider args"
  • opts: The list of options that the task requires/understands. The form of each option is {Key, $Character, "StringName", Spec, HelpText}, where:
    • Key is an atom, to be used to fetch the value later;
    • $Character is the short form of the option. So if the command is to be entered as a -c Arg, $c is the value of this field
    • Spec is either a type (atom, binary, boolean, float, integer, or string), a type with a default value ({Type, Val}), or the atom undefined.
  • profiles: Profiles to use for provider. Default to [default].
  • namespace: namespace the provider is registered in. Defaults to default, which is the main namespace.

These options are to be added to the provider when creating it.

A provider has the following implementation:

  1. -module(provider_template).
  2. -behaviour(provider).
  3. -export([init/1, do/1, format_error/1]).
  4. %% ===================================================================
  5. %% Public API
  6. %% ===================================================================
  7. %% Called when rebar3 first boots, before even parsing the arguments
  8. %% or commands to be run. Purely initiates the provider, and nothing
  9. %% else should be done here.
  10. -spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
  11. init(State) ->
  12. Provider = providers:create([Options]),
  13. {ok, rebar_state:add_provider(State, Provider)}.
  14. %% Run the code for the plugin. The command line argument are parsed
  15. %% and dependencies have been run.
  16. -spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.
  17. do(State) ->
  18. {ok, State}.
  19. %% When an exception is raised or a value returned as
  20. %% `{error, {?MODULE, Reason}}` will see the `format_error(Reason)`
  21. %% function called for them, so a string can be formatted explaining
  22. %% the issue.
  23. -spec format_error(any()) -> iolist().
  24. format_error(Reason) ->
  25. io_lib:format("~p", [Reason]).

List of Possible Dependencies

All dependencies are in the default namespace until indicated otherwise

NameFunctionProfileAlso depends on
app_discoveryExplore user applications and loads their configuration.default
cleanRemove compiled beam files from apps.defaultapp_discovery
compileCompile apps .app.src and .erl files.defaultlock
coverAnalyze cover-compiled filesdefaultlock
ctRun common test suites.testcompile
depsList dependencies.defaultapp_discovery
dialyzerRun the Dialyzer analyzer on the project.defaultcompile
edocGenerate documentation using edoc.defaultapp_discovery
eunitRun EUnit Tests.testcompile
helpDisplay a list of tasks or help for a given task or subtask.default
install_depsDownload dependenciesdefaultapp_discovery
lockLock dependencies and add rebar.lockdefaultinstall_deps
newCreate new project from templates.default
pkgsList available packages.default
releaseBuild release of project.defaultcompile
reportProvide a crash report to be sent to the rebar3 issues pagedefault
shellRun shell with project apps and deps in path.defaultcompile
tarTar archive of release built of project.defaultcompile
updateUpdate package index.default
upgradeUpgrade dependencies.default
versionPrint version for rebar and current Erlang.default
xrefRun cross reference analysisdefaultcompile

Note that you can depend on more than one provider, but they must be in the same namespace

Rebar API

Rebar comes with a module called rebar_api exporting commonly needed functions when writing providers. Functions include:

FunctionUsage
abort()Interrupts program flow
abort(FormatString, Args)Interrupts program flow; allows to display an ERROR message along with it. Equivalent to calling rebarapi:error(FormatString, Args) followed by rebar_api:abort()
console(FormatString, Args)Prints to the console.
info(FormatString, Args)Logs with the severity INFO
warn(FormatString, Args)Logs with the severity WARNING
error(FormatString, Args)Logs with the severity ERROR
debug(FormatString, Args)Logs with the severity DEBUG
expand_env_variable(InStr, VarName, RawVarValue)Given the env variable FOO, we want to expand all references to it in InStr. References can have two forms: $FOO and ${FOO}. The form $FOO is delimited by whitespace characters or the end of a line (eol).
get_arch()Returns the ‘architecture’ as a string of the form “$OTP_VSN-$SYSTEM$ARCH-WORDSIZE. Final strings will look like “17-x86_64-apple-darwin13.4.0-8” or “17-x86_64-unknown-linux-gnu-8”
wordsize()Returns the true wordsize of the emulator, i.e. the size of a pointer, in bytes as an string.
add_deps_to_path(RebarState)The project’s dependencies are added to the code path. Useful when a tool is invoked and needs to have global stateful access to libraries.
restore_code_path(RebarState)Revert the code path to only include the libraries required to run Rebar3 and its plugins. This is the desired state for Rebar3 to avoid conflicts with user-provided tools.
ssl_opts(Url)Returns the ssl options to use with httpc to make a secure and verified HTTP request.

Do note that all logging functions automatically add a new line (~n) to every expression logged.

Rebar State Manipulation

The State argument passed to the plugin provider can be operated on with the rebar_state module through the following interface:

FunctionUsage
get(State, Key, [DefaultValue]) -> ValueWhen a rebar.config element is of the form {Key, Value}., allows you to fetch the value for it
set(State, Key, Value) -> NewStateAdds a configuration value to the rebar state.
lock(State) -> ListOfLocksReturns a list of locked dependencies
escript_path(State) -> PathReturns the Rebar3 escript location
command_args(State) -> RawArgsReturns the arguments passed to rebar3
command_parsed_args(State) -> ArgsReturns the arguments passed to rebar3, parsed.
deps_names(State) -> DepsNameListReturns a list of dependencies’ names
project_apps(State) -> AppListReturns a list of applications. These can be handled using rebar_app_info.
all_deps(State) -> DepsListReturns a list of dependencies. These can be handled using rebar_app_info.
add_provider(State, Provider) -> NewStateRegisters a new provider, where Provider is the result of calling providers:create(Options). To be effective, this function must be called as part of a provider’s init/1 function. It can be called multiple times, allowing a plugin to register multiple commands.
add_resource(State, {Key, Module}) -> NewStateRegisters a new resource type (such as git, hg, and so on) with the module used to handle it. The resource must implement the rebar_resource behaviour. To be effective, this function must be called as part of a provider’s init/1 function.

Manipulate Application State

Each application being built (project applications and dependencies). All AppInfo records can be found in the State and accessed through project_apps/1 and all_deps/1

FunctionUsage
get(AppInfo, Key, [DefaultValue]) -> ValueFetch value of Key as defined for the application AppInfo
set(AppInfo, Key, Value) -> NewStateAdds a configuration value to the application’s record

Namespaces

For plugins that might require multiple commands all adapted to a single type of task (such as implementing a suite of tools for a BEAM language other than Erlang), rather than having multiple commands polluting the command space or requiring prefixes such as rebar3 mylang_compile, rebar3 introduces support for namespaces.

A plugin can be declared to belong to a given namespace. For example, the ErlyDTL compiler plugin introduces the compile command under the erlydtl namespace. It can therefore be invoked as rebar3 erlydtl compile. If the erlydtl namespace had other commands such as clean, they could be chained as rebar3 erlydtl clean, compile.

In other ways, a namespace acts like do (rebar3 do compile, edoc), but operating on a non-default set of commands.

To declare a namespace, an provider needs only to use the {namespace, Namespace} option in its configuration list. The provider will automatically register the new namespace and be available under this term.

🚧Namespaces also apply to provider dependencies and hooks

If a provider is part of a given namespace, its dependencies will be searched within that same namespace. Therefore if rebar3 mytool rebuild depends on compile, the compile command will be looked for in the mytool namespace.

To use the default compile command, the dependency must be declared as {default, compile}, or more generally {NameSpace, Command}.

The same mechanism is applied for hooks.

`

Tutorial

First version

In this tutorial, we’ll show how to start from scratch, and get a basic plugin written. The plugin will be quite simple: it will look for instances of TODO: lines in comments and report them as warnings. The final code for the plugin can be found on bitbucket.

The first step is to create a new OTP Application that will contain the plugin:

  1. rebar3 new plugin todo desc="example rebar3 plugin"
  2. ...
  3. cd todo
  4. git init
  5. Initialized empty Git repository in /Users/ferd/code/self/todo/.git/

The src/todo.erl file will be used to call call the initialization of all commands. For now we’ll only have one todo command. Open up the src/todo_prv.erl file that will contain the command implementation, and make sure you have the following skeleton in place:

  1. -module(todo_prv).
  2. -behaviour(provider).
  3. -export([init/1, do/1, format_error/1]).
  4. -define(PROVIDER, todo).
  5. -define(DEPS, [app_discovery]).
  6. %% ===================================================================
  7. %% Public API
  8. %% ===================================================================
  9. -spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
  10. init(State) ->
  11. Provider = providers:create([
  12. {name, ?PROVIDER}, % The 'user friendly' name of the task
  13. {module, ?MODULE}, % The module implementation of the task
  14. {bare, true}, % The task can be run by the user, always true
  15. {deps, ?DEPS}, % The list of dependencies
  16. {example, "rebar provider_todo"}, % How to use the plugin
  17. {opts, []} % list of options understood by the plugin
  18. {short_desc, "example rebar3 plugin"},
  19. {desc, ""}
  20. ]),
  21. {ok, rebar_state:add_provider(State, Provider)}.
  22. -spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.
  23. do(State) ->
  24. {ok, State}.
  25. -spec format_error(any()) -> iolist().
  26. format_error(Reason) ->
  27. io_lib:format("~p", [Reason]).

This shows all the basic content needed. Note that we leave the DEPS macro to the value app_discovery, used to mean that the plugin should at least find the project’s source code (excluding dependencies).

In this case, we need to change very little in init/1. Here’s the new provider description:

  1. Provider = providers:create([
  2. {name, ?PROVIDER}, % The 'user friendly' name of the task
  3. {module, ?MODULE}, % The module implementation of the task
  4. {bare, true}, % The task can be run by the user, always true
  5. {deps, ?DEPS}, % The list of dependencies
  6. {example, "rebar todo"}, % How to use the plugin
  7. {opts, []}, % list of options understood by the plugin
  8. {short_desc, "Reports TODOs in source code"},
  9. {desc, "Scans top-level application source and find "
  10. "instances of TODO: in commented out content "
  11. "to report it to the user."}
  12. ]),

Instead, most of the work will need to be done directly in do/1. We’ll use the rebar_state module to fetch all the applications we need. This can be done by calling the project_apps/1 function, which returns the list of the project’s top-level applications.

  1. do(State) ->
  2. lists:foreach(fun check_todo_app/1, rebar_state:project_apps(State)),
  3. {ok, State}.

This, on a high level, means that we’ll check each top-level app one at a time (there may often be more than one top-level application when working with releases)

The rest is filler code specific to the plugin, in charge of reading each app path, go read code in there, and find instances of ‘TODO:’ in comments in the code:

  1. check_todo_app(App) ->
  2. Paths = rebar_dir:src_dirs(rebar_app_info:opts(App), ["src"]),
  3. Mods = find_source_files(Paths),
  4. case lists:foldl(fun check_todo_mod/2, [], Mods) of
  5. [] -> ok;
  6. Instances -> display_todos(rebar_app_info:name(App), Instances)
  7. end.
  8. find_source_files(Paths) ->
  9. find_source_files(Paths, []).
  10. find_source_files([], Files) ->
  11. Files;
  12. find_source_files([Path | Rest], Files) ->
  13. find_source_files(Rest, [filename:join(Path, Mod) || Mod <- filelib:wildcard("*.erl", Path)] ++ Files).
  14. check_todo_mod(ModPath, Matches) ->
  15. {ok, Bin} = file:read_file(ModPath),
  16. case find_todo_lines(Bin) of
  17. [] -> Matches;
  18. Lines -> [{ModPath, Lines} | Matches]
  19. end.
  20. find_todo_lines(File) ->
  21. case re:run(File, "%+.*(TODO:.*)", [{capture, all_but_first, binary}, global, caseless]) of
  22. {match, DeepBins} -> lists:flatten(DeepBins);
  23. nomatch -> []
  24. end.
  25. display_todos(_, []) -> ok;
  26. display_todos(App, FileMatches) ->
  27. io:format("Application ~s~n",[App]),
  28. [begin
  29. io:format("\t~s~n",[Mod]),
  30. [io:format("\t ~s~n",[TODO]) || TODO <- TODOs]
  31. end || {Mod, TODOs} <- FileMatches],
  32. ok.

Just using io:format/2 to output is going to be fine.

To test the plugin, push it to a source repository somewhere. Pick one of your projects, and add something to the rebar.config:

  1. {plugins, [
  2. {todo, {git, "git@bitbucket.org:ferd/rebar3-todo-plugin.git", {branch, "master"}}}
  3. ]}.

Then you can just call it directly:

  1. rebar3 todo
  2. ===> Fetching todo
  3. ===> Compiling todo
  4. Application merklet
  5. /Users/ferd/code/self/merklet/src/merklet.erl
  6. todo: consider endianness for absolute portability

Rebar3 will download and install the plugin, and figure out when to run it. Once compiled, it can be run at any time again.

Optionally Search Deps

Let’s extend things a bit. Maybe from time to time (when cutting a release), we’d like to make sure none of our dependencies contain ‘TODO:’s either.

To do this, we’ll need to go parse command line arguments a bit, and change our execution model. The ?DEPS macro will now need to specify that the todo provider can only run after dependencies have been installed:

  1. -define(DEPS, [install_deps]).

We can add the option to the list we use to configure the provider in init/1:

  1. {opts, [ % list of options understood by the plugin
  2. {deps, $d, "deps", undefined, "also run against dependencies"}
  3. ]},

And then we can implement the switch to figure out what to search:

  1. do(State) ->
  2. Apps = case discovery_type(State) of
  3. project -> rebar_state:project_apps(State);
  4. deps -> rebar_state:project_apps(State) ++ lists:usort(rebar_state:all_deps(State))
  5. end,
  6. lists:foreach(fun check_todo_app/1, Apps),
  7. {ok, State}.
  8. [...]
  9. discovery_type(State) ->
  10. {Args, _} = rebar_state:command_parsed_args(State),
  11. case proplists:get_value(deps, Args) of
  12. undefined -> project;
  13. _ -> deps
  14. end.

The deps option is found using rebar_state:command_parsed_args(State), which will return a proplist of terms on the command-line after ‘todo’, and will take care of validating whether the flags are accepted or not. The rest can remain the same.

Push the new code for the plugin, and try it again on a project with dependencies:

  1. ===> Fetching todo
  2. ===> Compiling todo
  3. ===> Fetching bootstrap
  4. ===> Fetching file_monitor
  5. ===> Fetching recon
  6. [...]
  7. Application dirmon
  8. /Users/ferd/code/self/figsync/apps/dirmon/src/dirmon_tracker.erl
  9. TODO: Peeranha should expose the UUID from a node.
  10. Application meck
  11. /Users/ferd/code/self/figsync/_deps/meck/src/meck_proc.erl
  12. TODO: What to do here?
  13. TODO: What to do here?

Rebar3 will now go pick dependencies before running the plugin on there.

you can also see that the help will be completed for you:

  1. rebar3 help todo
  2. Scans top-level application source and find instances of TODO: in commented out content to report it to the user.
  3. Usage: rebar todo [-d]

That’s it, the todo plugin is now complete! It’s ready to ship and be included in other repositories.

Adding More Commands

To add more commands to the same plugin, simply add entries to the init function in the main module:

  1. -module(todo).
  2. -export([init/1]).
  3. -spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
  4. init(State) ->
  5. %% initialize all commands here
  6. {ok, State1} = todo_prv:init(State),
  7. {ok, State2} = todo_other_prv:init(State1),
  8. {ok, State2}.

And rebar3 will pick it up from there.