librefetch(8) -- downloads or creates a liberated source tarball
================================================================

## SYNOPSIS

`PKGBUILD:mksource()` is a Parabola extension to `PKGBUILD(5)`, and
`librefetch` is the program that implements that extension.

### most common use:

`makepkg --geninteg`

or a `makepkg --geninteg` wrapper, such as

`updpkgsums`

### download and create modes:

(this is usually done by `makepkg --geninteg`, not by a human)

`librefetch` [<OPTIONS>] <SOURCE-URL> [<OUTPUT-FILE>]

### alternate modes:

`librefetch` `-`[`g`\|`S`\|`M`\|`h`]

### PKGBUILD:

    source=("https://repo.parabola.nu/other/${pkgname}-libre/${pkgname}-${pkgver}.tar.zst"{,.sig})
    sha256sums(PRUNED_SOURCE_SUMS) # generated by `makepkg --geninteg`
    validpgpkeys=(PARABOLA_MAINTAINER_KEY)

    mksource=("https://UPSTREAM/${pkgname}-${pkgver}.tar.zst"{,.sig})
    mksha256sums=(UPSTREAM_SOURCE_SUMS) # generated by `librefetch --geninteg`
    mkvalidpgpkeys=(UPSTREAM_MAINTAINER_KEY)
    mkdepends=()
    mksource() {
      cd "$srcdir/$pkgname-$pkgver"
      rm the-non-free-parts
    }

## RATIONALE

Parabola is committed to not distributing non-free content, and so
when an upstream source tarball contains non-free parts, it isn't
adequate to delete those parts in `PKGBUILD:prepare()`; we'd still be
distributing the non-free parts in the `.src.tar.zst` files.
`PKGBUILD:mksource()` is our solution; it is a function that can prune
the sources before building the `.src.tar` file.

Note that it isn't necessary to use `mksource()` instead of
`prepare()` for all FSDG violations, just cases where there is actual
non-free content.  Other types of FSDG violations (steering users
toward non-free software, uses-non-free, et c.) are better handled in
`prepare()`.

## DESCRIPTION

`librefetch` is a "download" agent that sometimes creates files
locally instead of downloading them.  Most of the time, librefetch
simply calls `curl` (or rather: `librefetch.conf(5)`:<DOWNLOADER>),
but if the URL starts with a location that Parabola uploads "custom"
source tarballs to (`librefetch.conf(5)`:<MIRRORS>) and no file is at
that URL, then librefetch will automatically create it for you.

The drop-in config file `/etc/makepkg.conf.d/librefetch.conf`
configures `makepkg(8)` to use `librefetch` instead of `curl(1)` for
`https://` URLs (and, for backwards-compatibility, `libre://` URLs).
Because of this, it is almost never necessary to call `librefetch`
manually; `makepkg --geninteg` (and wrappers such as `updpkgsums(8)`)
will call `librefetch` for you.

`librefetch` has 6 modes of operation:

 * `download`: Download the tarball from the configured mirror.
 * `create`: Create the tarball from a `PKGBUILD`/`SRCBUILD`.
 * `checksums`: Generate integrity checks for source files.
 * `srcbuild`: Print the effective build script.
 * `makepkg`: Generate and print the location of the effective makepkg
   script.
 * `help`: Print `librefetch` usage information.

The default mode of operation is `download` mode.  If `download` mode
fails, it may choose to try `create` mode.

## OPTIONS

### SETTINGS FOR DOWNLOAD AND CREATE MODES

 * `-C`: Force `create` mode (don't download); still requires a URL to
   be given.
 * `-D`: Force `download` mode (don't create)
 * `-p` <file>: Use an alternate build script for `create` mode
   (instead of `PKGBUILD`).  If an `SRCBUILD` file exists in the same
   directory, it is used instead.
 * Any options documented in `makepkg -h` are accepted and in `create`
   mode are passed to the modified copy of makepkg.

### ALTERNATE MODES

Other modes do not accept any positional arguments (no URL should be
given).

 * `-g`, `--geninteg`: Use `checksums` mode: Generate integrity checks
   for source files.
 * `-S`, `--srcbuild`: Use `srcbuild` mode: Print the effective build
   script.
 * `-M`, `--makepkg`: Use `makepkg` mode: Generate and print the
   location of the effective makepkg script.
 * `-h`, `--help`: Use `help` mode: Show usage information.

## DOWNLOAD MODE

If <SOURCE-URL> begins with the string `libre://`, it is replaced with
the first value in `librefetch.conf(5)`:<MIRRORS>; this is for
compatibility with `PKGBUILD` files that used a previous version of
librefetch.

librefetch uses `librefetch.conf(5)`:<DOWNLOADER> to attempt to
download the source tarball from that URL.  If that fails, and
following conditions are met, it proceeds to `create` mode:

 * The `-D` flag has not been specified to inhibit `create` mode.
 * The <SOURCE-URL> begins with one of the values in
   `librefetch.conf(5)`:<MIRRORS>.

The latter requirement allows librefetch to be used as a generic
HTTP/HTTPS download agent, that can automatically create files from
whitelisted locations.

## CREATE MODE

The principle of `create` mode is that a special `PKGBUILD(5)` (called
`SRCBUILD`) installs source files to <$pkgdir>, and the resulting
"package" is then used as a source tarball.  The `SRCBUILD` exists in
the same directory as the `PKGBUILD`.  It can be created manually, or
generated on-the-fly from the `PKGBUILD`.  Extra steps are taken to
ensure that as long as the same directory contents go in, an identical
tarball will come out--the checksum of the file should not change
based on when it is built or who builds it.

The `SRCBUILD` is either created, or sanitized if it already exists.
If the output filename does not end with `.sig` or `.sig.part`, then
the `SRCBUILD` is fed to a modified version of `makepkg(8)`.  If the
output filename does end with `.sig` or `.sig.part`, then it uses GPG
to create a signature.  If the file it is trying to sign does not
exist yet, librefetch recurses on itself to create it.

The reason `makepkg` must be modified is that we need the resulting
tarball to be deterministic (as well as not containing package
metadata).

When this documentation speaks of a file being modified, it is a
temporary copy of the file that is modified, your original file will
remain intact.

## SRCBUILD GENERATION

As explained in the `CREATE MODE` section, in `create` mode, this
program generates an `SRCBUILD` file.  For debugging purposes, this
file can be printed instead of executed with `srcbuild` mode.

### PRE-EXISTING SRCBUILD

The use of `SRCBUILD` files predates `librefetch`.  By convention,
they set <PKGDEST> and <PKGEXT> in `package()` in order to modify the
behavior of `makepkg`.  Because a modified version of `makepkg` is
used, this interferes with the correct behavior.  To compensate for
this, lines containing "`PKGDEST=`" or "`PKGEXT=`" are deleted from
the `SRCBUILD`.

The general idea is that `build()` makes any modifications to the
source, then `package()` copies it from <$srcdir> to <$pkgdir>.

### SRCBUILD FROM PKGBUILD:mksource()

Possibly more elegant than having a separate `SRCBUILD` file is having
an `mksource()` function in the main `PKGBUILD`.  This results in less
boilerplate and fewer files to edit.

Note that this only happens if a file named `SRCBUILD` doesn't already
exist; when migrating a package from a manually created `SRCBUILD` to
this method, the `SRCBUILD` must be deleted (or renamed) for this to
work.

The dynamically created `SRCBUILD` is created by copying `PKGBUILD` to
a temporary file, then re-setting variables and re-defining functions.
Following is a table of the translations.

    Variables
      source       = mksource
      noextract    = mknoextract
      validpgpkeys = mkvalidpgpkeys
      *sums        = mk*sums (md5, sha1, sha224, sha256, sha384, sha512)
      depends      = <empty>
      checkdepends = <empty>
      makedepends  = mkdepends
      *_$CARCH     = <unset>
      backup       = <empty>
      install      = <unset>
    Functions
      prepare() { :; }
      build()   { mksource; }
      check()   { :; }
      package() { cp -a "$srcdir"/*/ "$pkgdir/"; }

The `mksource()` function does not need to be defined.  If it isn't
defined, then no transformations will be made to the source between it
being extracted to <$srcdir> and copied to <$pkgdir>.  Before makepkg
supported VCS sources, this was useful for creating tarballs of VCS
checkouts... I'm not sure it's useful anymore.

In summary:

 * Set <mksource=()> and <mkmd5sums=()> to act as <source=()> and
   <md5sums=()>, respectively.
 * Declare a `mksource()` function to make modifications to the
   source, if necessary.

Other changes:

 * <pkgname> is set to <pkgbase>, or the first element of the
   <pkgname> array (the effect is that split packaging is turned
   off).
 * <options=()> is set have `makepkg` avoid making changes to
   <$pkgdir>.  The exact change is:

      options=(!strip docs libtool staticlibs emptydirs !zipman !debug purge)

 * <PURGE_TARGETS=()> has VCS directories added to it:

      PURGE_TARGETS=(.bzr/ .cvs/ .git/ .hg/ .svn/ .makepkg/)

## MAKEPKG MODIFICATIONS

The following modifications are made to makepkg:

 * Allow us to manipulate the output file (<$pkg_file>)
 * Do not include metadata in the output file (`.PKGINFO`,
   `.BUILDINFO`, and `.MTREE`)
 * Force 'ustar' tar format, don't allow it to upgrade to 'pax' to
   store extended file attributes.
 * Don't symlink the resulting file into the current directory.
 * <PURGE_TARGETS> interprets an item as a directory if it ends with a
   slash ("/").
 * Timestamps in <$pkgdir> are reset to the date specified in
   <SOURCE_DATE_EPOCH>[1], or "1990-01-01 0:0:0 +0" if it's not set, so
   that the resulting tarball will be the same, regardless of when it
   was created.
 * Sort the files included in the tarball; normally the order of files
   in a tarball is essentially random (even if it tends to be the same
   when re-created on the same machine).
 * append `-libre` to <$srcdir>
 * append `-libre` to <$pkgbasedir> (which becomes <$pkgdir>)
 * Don't check if the package has already been built.
 * Have logfiles be named
   `${pkgbase}-${fullver%-*}-mksource-${pkgfunc}.log` instead of
   `${pkgbase}-${fullver}-${CARCH}-${pkgfunc}.log`.

For debugging purposes, this modified makepkg can be printed instead
of executed with `makepkg` mode.  Before it is run in create mode,
`PKGEXT`, `PKGDEST`, and `pkg_file` are set as environment variables.

## CONFIGURATION

See `librefetch.conf(5)` for details on configuring librefetch using
the `librefetch.conf` file.

Be aware that overriding `DLAGENTS` in `~/.config/pacman/makepkg.conf`
or `~/.makepkg.conf` can disable `makepkg` from running `librefetch`.

## HISTORY

In the bad old days (2009-2013), Parabola had two conventions for
liberating sources:

 1. Write a `PKGBUILD(5)` named `SRCBUILD` that when built with
    `makepkg -p SRCBUILD` builds a "package" containing pruned sources
    rather than a built package.  The maintainer would run `makepkg -p
    SRCBUILD` then rename the resulting file from `.pkg.tar.gz` to
    `.tar.gz`, upload it somewhere, and point the main `PKGBUILD` at
    that.

 2. Add a `mksource()` function to the main `PKGBUILD` that would
    prune the sources.  The maintainer would `source PKGBUILD` in
    their interactive shell, then manually run the `mksource` function
    to prune the sources.  They'd tar that up, upload it somewhere,
    and point the main `PKGBUILD` ad that.

Both of these were bad: it was a lot of manual work and things tended
to get out of sync, and it wasn't very consistent exactly which things
need to be duplicated (Should the `mksource()` function download the
upstream sources, or is that manual?  Should it tar things up
for you, or is is that manual?).

So, in 2013 we sought to streamline and standardize this, and invented
`librefetch`.

The normal package-update flow tends to look something like:

 1. Update `PKGBUILD:pkgver`
 2. Run `makepkg --geninteg` (or `updpkgsums`)
 3. Run `sudo libremakepkg`
 4. Run `librestage`
 5. Run `librerelease`

We wanted to keep the same workflow for packages that we
have to produce pruned source tarballs for.  There are 3 parts to
this solution:

 - Hook into `makepkg.conf(5)`:<DLAGENTS> so that for when it sees a
   file in the `PKGBUILD`:<source> array that starts with
   `https://repo.parabola.nu/other/` it uses a new `librefetch`
   command to "fetch" that file, instead of using `curl` to fetch it.
 - Enhance `librestage` to stage files in the `PKGBUILD`:<source>
   array that start with `https://repo.parabola.nu/other/`.
 - Enhance `librerelease` be upload those files without calling
   `db-update` on them.

The `librestage` and `librerelease` enhancements have turned out to be
handy even for files that aren't created by `librefetch`; for example,
it's now how we upload the libretools source tarballs.

At the same time, we didn't want to force maintainers of packages with
existing `SRCBUILD` files or `mksource()` functions to rewrite their
files, so we tried to made `librefetch` accept what folks already had
written.

Just a few months after we first released librefetch (2013-01-17),
makepkg 4.1.0 (2013-04-01) added both the `PKGBUILD:prepare()`
function, and support for VCS <source> URLs.  At first we were
excited, perhaps we could use this instead of `mksource()` and get rid
of `librefetch` (the best code is no code!).  However, we determined
that while VCS <source> URLs are good enough for cases that don't
require removing non-free content, `prepare()` was not adequate for
removing non-free content, as we would still be distributing the
non-free content in our `.src.tar` files.

## SEE ALSO

libretools: librefetch.conf(5)

pacman: PKGBUILD(5), makepkg(8), makepkg.conf(5)

pacman-contrib: updpkgsums(8)

curl: curl(1)

## NOTES

 1. <SOURCE_DATE_EPOCH> specification for build systems
    https://reproducible-builds.org/specs/source-date-epoch/

<!--
  Copyright (C) 2013-2016, 2018, 2024  Luke T. Shumaker <lukeshu@parabola.nu>

  SPDX-License-Identifier: GPL-2.0-or-later

  This file is part of Parabola Libretools.

  Libretools is free software: you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation, either version 2 of the License, or
  (at your option) any later version.

  Libretools is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-->
