Jekyll2021-01-10T12:30:05+00:00https://blog.sebnow.com/feed.xmlslogThoughts about junk and stuff, or, whatever.Sebastian NowickiProvisioning a Raspberry Pi Swarm with IaC: Part 12020-11-12T00:00:00+00:002020-11-12T00:00:00+00:00https://blog.sebnow.com/posts/2020/11/12/Provisioning-a-Raspberry-Pi-Swarm-with-IaC-Part-1<p>The Raspberry Pi is a cheap and fun piece of kit for geeks. My QNAP NAS
is running a few too many applications on Docker, and I have a few
Raspberry Pi’s laying around, so I thought it’d be a fun weekend project
to turn them into a Docker Swarm.</p>
<h2 id="the-swarm">The Swarm</h2>
<p>Before getting into the automation side of things, let’s set up
a prototype of the swarm itself. I’ll be setting up;</p>
<ul>
<li>A Raspberry Pi Rev 1 Model B,</li>
<li>A Raspberry Pi Rev 3 Model B,</li>
</ul>
<p>The Raspberry Pi’s will be running <a href="https://www.raspberrypi.org/software/operating-systems/">Raspberry Pi OS</a> Buster
(2020-08-20). While the Rev 3 is 64 bit, I’ll be using the <code class="language-plaintext highlighter-rouge">armhf</code> on
both. All the machines are on the same network.</p>
<p>Follow the official <a href="https://www.raspberrypi.org/documentation/installation/installing-images/">instructions</a> to burn the image
to the SD cards and spin up the Pi’s. We’ll be needing Docker, so follow
<a href="https://docs.docker.com/engine/install/debian/">the instructions</a> for installing that too.</p>
<p>The Docker Swarm has two types of nodes; managers and workers. The
managers keep the state of the swarm in a distributed store by using the
<a href="https://docs.docker.com/engine/swarm/raft/">Raft Consensus Algorithm</a>. For best results, the amount of
managers should be odd. I only have three machines to work with, so I’ll
stick with a single manager for now. This cluster doesn’t need to be
highly available. The Rev 3 is the most powerful RPi, so I’ll be
electing it as the manager.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker swarm init
Swarm initialized: current node (eseiud89bm9gl3eail1tnnznx) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-6814zqm6wpp4q43fe9a9m2ezxcsecjnlmvcayvm5ydzpwkegdh-4n5qaw4ujge9xcmi2vcr0mcrn 192.168.0.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
eseiud89bm9gl3eail1tnnznx * manager Ready Active Leader 19.03.13
</code></pre></div></div>
<p>Docker is nice enough to provide instructions for the workers. Let’s do
that.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@worker:~$ sudo docker swarm join --token SWMTKN-1-6814zqm6wpp4q43fe9a9m2ezxcsecjnlmvcayvm5ydzpwkegdh-4n5qaw4ujge9xcmi2vcr0mcrn 192.168.0.2:2377
This node joined a swarm as a worker.
</code></pre></div></div>
<p>All management commands need to be run on the manager, so let’s check
the list of nodes;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
eseiud89bm9gl3eail1tnnznx * manager Ready Active Leader 19.03.13
bcm9cnme7ewvx224klfun1qfn worker Ready Active 19.03.13
</code></pre></div></div>
<p>Now let’s run an application on the swarm. For demonstration purposes
let’s just use some random image, like <code class="language-plaintext highlighter-rouge">nginxdemos/hello</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker pull nginxdemos/hello
Using default tag: latest
latest: Pulling from nginxdemos/hello
550fe1bea624: Pull complete
d421ba34525b: Pull complete
fdcbcb327323: Pull complete
bfbcec2fc4d5: Pull complete
0497d4d5654f: Pull complete
f9518aaa159c: Pull complete
a70e975849d8: Pull complete
Digest: sha256:f5a0b2a5fe9af497c4a7c186ef6412bb91ff19d39d6ac24a4997eaed2b0bb334
Status: Downloaded newer image for nginxdemos/hello:latest
docker.io/nginxdemos/hello:latest
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker service create -p 8080:80 nginxdemos/hello:0.2
image nginxdemos/hello:0.2 could not be accessed on a registry to record
its digest. Each node will access nginxdemos/hello:0.2 independently,
possibly leading to different nodes running different
versions of the image.
9ldqvus6y2zzp64pj187mmb52
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
9ldqvus6y2zz jolly_rhodes replicated 1/1 nginxdemos/hello:0.2 *:8080->80/tcp
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ sudo docker service ps jolly_rhodes
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
hfebnhjrg0v7 jolly_rhodes.1 nginxdemos/hello:0.2 manager Running Running 7 minutes ago
</code></pre></div></div>
<p>Fantastic. We have a service running in the swarm. Let’s see if it
works;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pi@manager:~$ links -dump http://localhost:8080
NGINX Logo
Server address: 10.0.0.35:80
Server name: 2c704cbf842b
Date: 12/Nov/2020:19:53:43 +0000
URI: /
[ ] Auto Refresh
Request ID: e620aa94f6e13add9a69cd03340b1551
© NGINX, Inc. 2018
</code></pre></div></div>
<p>Now you can do all the typical things with Docker Swarm; adding
replicas, deploying stacks and adding more nodes. In the next post we’ll
go through how this can be automated using Ansible to easily scale to
more nodes.</p>Sebastian NowickiThe Raspberry Pi is a cheap and fun piece of kit for geeks. My QNAP NAS is running a few too many applications on Docker, and I have a few Raspberry Pi’s laying around, so I thought it’d be a fun weekend project to turn them into a Docker Swarm.A Clearer View of the NeoVim Ecosystem2020-10-10T00:00:00+00:002020-10-10T00:00:00+00:00https://blog.sebnow.com/posts/2020/10/10/A-Clearer-View-of-the-NeoVim-Ecosystem<p>A while back I posted about <a href="/posts/2020/05/13/NeoVim-A-Clean-Slate">rewriting my Vim config from scratch</a> and focusing on using
Neovim. In that time, my assumptions about what plugins I’d be using
have changed, as I have discovered more plugins that are implemented in
Lua and depend on NeoVim 0.5+.</p>
<h2 id="completion">Completion</h2>
<p>I checked out <a href="https://github.com/nvim-lua/completion-nvim">completion-nvim</a> when I first looked into using the
built in LSP client, however it seemed to lack a few features I wanted,
like multiple sources. <a href="https://github.com/ncm2/ncm2">ncm2</a> was better at the time. Since then,
<code class="language-plaintext highlighter-rouge">completion-nvim</code> has added a slew of features to be on par with <code class="language-plaintext highlighter-rouge">ncm2</code>,
at least for me.</p>
<h2 id="fuzzy-search">Fuzzy Search</h2>
<p>Instead of <a href="https://github.com/Shougo/denite.nvim">denite</a>, I will now be looking into customizing
<a href="https://github.com/nvim-lua/telescope.nvim">telescope.nvim</a>. It appears to provide similar functionality but is
written in Lua, which makes it faster than a Python implementation, and
for me makes it more hackable. It has LSP integration out of the box, so
that reduces the amount of work I have to do.</p>
<p>I’m currently using it as a fuzzy file finder (CtrlP/fzf replacement),
a buffer list/switcher, LSP references, and grep/rg.</p>
<h2 id="treesitter">TreeSitter</h2>
<p>Along with LSP and Lua, the most exciting feature being developed in
NeoVim is <a href="https://tree-sitter.github.io/">TreeSitter</a> integration. TreeSitter is an incremental
parser generator, initially used in Atom and GitHub. It allows NeoVim to
work on an actual abstract syntax tree instead of relying on regular
expressions. Highlighting can be more accurate, reliable,
and tolerant of errors. Code can be navigated contextually. Folds can
work on custom parser queries. A lot of cool stuff is made possible by
this. Max Brunsfeld, the author, has an excellent
<a href="https://www.youtube.com/watch?v=Jes3bD6P0To">presentation</a> from
StrangeLoop 2018.</p>
<h2 id="plugin-management">Plugin Management</h2>
<p>While I haven’t started using it yet, <a href="https://github.com/wbthomason/packer.nvim">Packer</a> looks like an
interesting replacement for <code class="language-plaintext highlighter-rouge">vim-plug</code>. Support for dependencies could
be a useful feature not available in <code class="language-plaintext highlighter-rouge">vim-plug</code>.</p>Sebastian NowickiA while back I posted about rewriting my Vim config from scratch and focusing on using Neovim. In that time, my assumptions about what plugins I’d be using have changed, as I have discovered more plugins that are implemented in Lua and depend on NeoVim 0.5+.NeoVim - A Clean Slate2020-05-13T00:00:00+00:002020-05-13T00:00:00+00:00https://blog.sebnow.com/posts/2020/05/13/NeoVim-A-Clean-Slate<p>According to <a href="https://github.com/sebnow/configs/blob/b3dccd4a5dc025285f7ad2bee12bf7d4144b2104/.vimrc">Git</a>, my vimrc is twelve years old, and
it’s probably actually older. That’s a long time to accrue a lot of
cruft. A lot has happened in the world of Vim in the past decade, with
the emergence of <a href="https://neovim.io">NeoVim</a> encouraging the development of
features found in newer editors. Simultaneously, tooling for languages
has improved significantly with the creation of the <a href="https://microsoft.github.io/language-server-protocol">Language Server
Protocol</a> (LSP), and newer languages, such as Rust, learning from
their predecessors and providing tooling out of the box. I think it’s
time to wipe the slate clean and start anew, looking for modern
solutions to configure the editor the way I want it to work.</p>
<p>My main goal is to have functionality rivaling that of “modern” editors
and IDEs, while having access to the modes, keybindings and performance
of Vim. NeoVim 0.5 provides a fantastic foundation for achieving this,
with a native LSP client, Lua support, a built in terminal, and
asynchronous jobs.</p>
<p>To start off, I’ve removed all but the most <a href="https://github.com/sebnow/configs/blob/0a826279e957ca07b107bdb7563237ad133c10dc/vim/settings.vim">minimal
settings</a> to make NeoVim usable for me.</p>
<p>The ecosystem around the LSP client is still maturing, as 0.5 is yet to
be released, but it’s starting to become viable as a daily driver.</p>
<ul>
<li><a href="https://github.com/ncm2/ncm2">ncm2</a> already includes support for it, which provides
completion-as-you-type for multiple sources (LSP, snippets,
omnicomplete).</li>
<li><a href="https://github.com/haorenW1025/diagnostic-nvim">diagnostic-nvim</a> improves the user experience by delaying
diagnostics while in insert mode - in most cases you don’t end up with
a well-formed program with each key press - and utilizing floating
windows to present information in a more readable manner.</li>
<li><a href="https://github.com/weilbith/nvim-lsp-denite">nvim-lsp-denite</a> providing integration with Denite.</li>
</ul>
<p>Currently, the most mature alternative to the NeoVim LSP client is
<a href="https://github.com/neoclide/coc.nvim">CoC</a>, which provides a great user experience, but tries to do too
much for my taste. It appears to be providing a bridge to VSCode
extensions.</p>
<p>I’m looking at using <a href="https://github.com/Shougo/denite.nvim">Denite</a> as a unified interface for quickly
navigating things like Code Actions, files, buffers, and git. It may
also serve as a platform to run common tasks such as running tests in
a consistent way. I haven’t used it previously, mostly due to the
dependency on Python, but using it will remove the need for other
plugins such as <a href="https://github.com/kien/ctrlp.vim">ctrlp.vim</a>, <a href="https://github.com/junegunn/fzf.vim">fzf.vim</a>, <a href="https://github.com/liuchengxu/vim-clap">vim-clap</a>, etc.</p>
<p>I will be elaborating on specific parts of my set up in future posts.</p>Sebastian NowickiAccording to Git, my vimrc is twelve years old, and it’s probably actually older. That’s a long time to accrue a lot of cruft. A lot has happened in the world of Vim in the past decade, with the emergence of NeoVim encouraging the development of features found in newer editors. Simultaneously, tooling for languages has improved significantly with the creation of the Language Server Protocol (LSP), and newer languages, such as Rust, learning from their predecessors and providing tooling out of the box. I think it’s time to wipe the slate clean and start anew, looking for modern solutions to configure the editor the way I want it to work.Lambda Days 2017 - The First Day2017-02-10T00:00:00+00:002017-02-10T00:00:00+00:00https://blog.sebnow.com/posts/2017/02/10/Lambda-Days-2017-The-First-Day<p>The day kicked off with a Keynote from John Hughes, who provided an
energetic introduction to functional programming, recapped some of its
history, and explained <a href="http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html">why it matters</a>, along with Mary Sheeran.</p>
<p>Perhaps the most surprising talk was by Sydney Padua, a cartoonist and
animator - an unlikely presenter for a Computer Science related
conference. She talked about how a one-off comic, portraying Charles
Babbage and Ada Lovelace, turned into a series, and eventually evolved
into a very informative graphic novel.</p>
<p><img src="https://blog.sebnow.com/assets/sydney_padua_econ2_008.jpg" alt="Analytical Engine" />
<em><a href="http://sydneypadua.com/2dgoggles/lovelace-and-babbage-vs-the-economy-pt-2/">Lovelace and Babbage Vs. The Economy, Pt 2</a></em></p>
<p>As Padua made more comics, she learned more about the Babbage’s
Difference and Analytical engines. While a working Difference Engine
exists, in the Computer History Museum in Mountain View, CA, there is no
Analytical Engine to use as a reference. Padua ended up recreating the
Analytical Engine in 3D Modelling software using technical diagrams as
reference, and even animated parts of the engine. Seeing the intricate
detail and huge scale of the engine was extremely impressive, especially
considering how little memory and computation power it would have had,
compared to modern technology.</p>
<p>With a majority of the day left, it was time to delve into some
technical talks. Unfortunately these were found wanting. Perhaps my
choice of tracks was poor, as I avoided the Research track for fear of
everything going over my head. The talks I attended were mostly a mix of
basic FP concepts or marketing/recruitment material under the veil of
a “How we solved X” type presentation.</p>
<p>There were a few talks which stood out, such as “Free the Conqueror!” by
Tamás Kozsik, “Implementing Event-Driven Microservices Architecture
using Functional programming” by Nikhil Barthwal, and “Keeping the flow
going: Data-flow oriented workflow systems” by Annette Bieniusa.</p>
<p>Tamás Kozsik discussed a tool he is working on which refactors Divide
and Conquer algorithms to be better suited for parallelisation, using
static analysis. This is done by normalising such functions into five
distinct parts;</p>
<ol>
<li>Check if the current element is the base case,</li>
<li>If so, solve the case,</li>
<li>Otherwise divide the search space</li>
<li>Map the solve function over each half,</li>
<li>Combine the results</li>
</ol>
<p>This normalised form lends itself to easy parallelisation as each leaf
can be solved in parallel.</p>
<p>Microservices are all the rage at the moment, and it seems FP is gaining
traction. Nikhil Barthwal discussed how <a href="http://www.jet.com">Jet.com</a>
combined the two concepts, building their infrastructure completely on
microservices and F#. He drew parallels between FP and microservices,
explaining that there can be pure and impure microservices (loosely
speaking, e.g., logging isn’t considered impure), microservices
typically don’t have state, but rather react to events - similar to how
immutable data is a collection of states. These features makes
parallelising microservices fairly straight forward. Unfortunately he
didn’t elaborate much more than that.</p>
<p>In her talk, Annette Bieniusa described a method of modelling tasks
(forms and actions) using Kleisli Arrows, which <a href="http://www.cse.chalmers.se/~rjmh/afp-arrows.pdf">John Hughes discussed in
a paper</a>. Not having been around academia much, it was
interesting to hear John Hughes ask questions about the concept.</p>
<p>Attending <a href="http://www.lambdadays.org/lambdadays2017">Lambda Days 2017</a> has reminded me how amazing
Functional Programming is. Listening to some iconic people in the
field, such as John Hughes, was really inspiring. The amount of
attendees was comforting, showing how popular the paradigm is becoming.
If you’re not writing applications in a functional language, you’re more
than likely using a significant amount of FP features regardless.</p>Sebastian NowickiThe day kicked off with a Keynote from John Hughes, who provided an energetic introduction to functional programming, recapped some of its history, and explained why it matters, along with Mary Sheeran.A Universal Package Specification2009-12-05T00:00:00+00:002009-12-05T00:00:00+00:00https://blog.sebnow.com/posts/2009/12/05/A-Universal-Package-Specification<p>One of my favourite things about Unix-like operating systems is package
management. They have many technical advantages such as:</p>
<ul>
<li>ease of installation - just search, install,</li>
<li>automatic dependency resolution,</li>
<li>reduction in storage usage by utilizing shared libraries (no <a href="http://en.wikipedia.org/wiki/DLL_hell/">DLL hell</a>),</li>
<li>clean filesystem,</li>
<li>etc.</li>
</ul>
<p>It also has many practical advantages such as listing installed packages,
showing package information (version, url), and finding which package a file
belongs to. Unfortunately the implementation of this concept isn’t always
perfect. For instance, the learning curve for <a href="http://www.debian.org/doc/debian-policy/">Debian packaging</a> is quite
steep, and RPM has often been criticised for <a href="http://en.wikipedia.org/wiki/Dependency_hell/">dependency hell</a>.</p>
<p>When I first looked at <a href="http://www.archlinux.org/">Archlinux</a>’s package management system, I was
pleasantly surprised at how simple the <a href="http://www.archlinux.org/pacman/PKGBUILD.5.html">PKGBUILD</a> format is. It is a shell
script, in which variables define metadata, and a <code class="language-plaintext highlighter-rouge">build()</code> function
specifies the steps required to build the package.</p>
<h2 id="problems">Problems</h2>
<p>The <a href="http://aur.archlinux.org/">Archlinux User Repository</a> allows users to contribute PKGBUILDs
without a lengthy review process. The code behind it wasn’t in the greatest
shape, so I decide to rewrite it in Python + Django. During development I ran
into the problem of parsing the PKGBUILD format. I had to get the metadata
into the database somehow. Since a PKGBUILD is just a shell script, I thought
sourcing it and outputting Python, then eval-ing it would be the easiest way
of doing it. Unfortunately this has many security problems, including the
execution of malicious code, or infinite loops (the web server would hang). I
was forced to write a minimal <a href="http://www.github.com/sebnow/pkgparse/">shell parser</a> to extract the
metadata. While it removes the security concerns, it raises others, such as
inaccuracy and maintenance problems (the specification and parser code is
tightly coupled). It turns out the <a href="http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_10">Shell grammar</a> isn’t exactly the simplest
one.</p>
<p>The PKGBUILD format has some warts, which are mostly due to the lack of data
structures in bash, such as hash tables. For instance there are two arrays,
<code class="language-plaintext highlighter-rouge">source</code> and <code class="language-plaintext highlighter-rouge">md5sums</code>. Each element in the <code class="language-plaintext highlighter-rouge">md5sums</code> array maps to the
respective <code class="language-plaintext highlighter-rouge">source</code> element. As you can imagine this can easily result in
bugs. Fortunately <code class="language-plaintext highlighter-rouge">makepkg</code> (the package creation utility) is able to create
this field automatically.</p>
<p>Another problem associated with <code class="language-plaintext highlighter-rouge">sources</code> and checksums is with binary
sources. A binary source is typically different for each supported
architecture. There is no way of specifying this easily in the PKGBUILD
format. A common solution is to do something like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">arch</span><span class="o">=(</span><span class="s1">'i686'</span> <span class="s1">'x86_64'</span><span class="o">)</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$CARCH</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"x86_64"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">source</span><span class="o">=(</span><span class="s2">"http://foo.com/</span><span class="k">${</span><span class="nv">pkgname</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">pkgver</span><span class="k">}</span><span class="s2">-x86_64.bin"</span><span class="o">)</span>
<span class="nv">md5sums</span><span class="o">=(</span>4dacc4474e93bcd4e168fdc48c4e6aee<span class="o">)</span>
<span class="k">else
</span><span class="nb">source</span><span class="o">=(</span><span class="s2">"http://foo.com/</span><span class="k">${</span><span class="nv">pkgname</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">pkgver</span><span class="k">}</span><span class="s2">-i386.bin"</span><span class="o">)</span>
<span class="nv">md5sums</span><span class="o">=(</span>5001378e4f83d0d6db014eec9182f7b4<span class="o">)</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>The checksum generation feature of <code class="language-plaintext highlighter-rouge">makepkg</code> no longer works properly, and in
order to parse this metadata, an interpreter is now required, not just a
parser.</p>
<h2 id="solutions">Solutions</h2>
<p>An Archlinux user started defining <a href="http://xyne.archlinux.ca/ideas/pkgmeta/">an alternate PKGBUILD
specification</a>. It addresses some problems with the shell format,
such as extendability (to a degree) and ease of parsing. Unfortunately this
format is a completely new data format, and thus requires a parser of its
own.</p>
<p>Lately I’ve been toying with the idea of creating a universal package
specification. The idea of this specification is to provide a portable
way of defining package metadata, while keeping it simple, and extendable.
Ideally any package manager would be able to use this format, and have enough
metadata to do what they need to. It is extendable with an <code class="language-plaintext highlighter-rouge">extensions</code>
field, which allows package managers to get any data they require, which is
not already included in the specification. If there is enough demand for an
extension, it should be added to the next revision of the specification.</p>
<p>A common data serialization format is <a href="http://www.yaml.org/">YAML</a>. It is simple, easy to parse, and
very versatile. For these reasons it was my first choice for the
specification. There are already many different parsers in many different
languages. Thus, the format should be easily accessible in most languages.
Unfortunately it does not seem that bash is one of them, so parts of
<code class="language-plaintext highlighter-rouge">makepkg</code> would have to be rewritten.</p>
<h2 id="suggested-format">Suggested Format</h2>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">foo</span>
<span class="na">version</span><span class="pi">:</span> <span class="m">1.0</span>
<span class="na">release</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">summary</span><span class="pi">:</span> <span class="s">A fictional package to show the YAML package format</span>
<span class="na">description</span><span class="pi">:</span> <span class="pi">></span>
<span class="s">This package is an example of the YAML universal package format, which</span>
<span class="s">aims to be portable and extandable. This description can be as long as</span>
<span class="s">it needs to be.</span>
<span class="na">architectures</span><span class="pi">:</span> <span class="c1"># Keys denote architectures supported by this package</span>
<span class="na">any</span><span class="pi">:</span>
<span class="na">sources</span><span class="pi">:</span> <span class="c1"># URIs of source files, such as a source tarball</span>
<span class="pi">-</span> <span class="na">uri</span><span class="pi">:</span> <span class="s">http://www.foo.org/-.tar.gz</span>
<span class="na">checksums</span><span class="pi">:</span> <span class="c1"># keys denote algorithm, values are the checksums</span>
<span class="na">md5</span><span class="pi">:</span> <span class="s">fd085a845298afb36f6676feac855e63</span>
<span class="na">sha1</span><span class="pi">:</span> <span class="s">e19f73b340aeae43b98908006094af62e0c7b5b9</span>
<span class="pi">-</span> <span class="na">uri</span><span class="pi">:</span> <span class="s">shared_data-.tar.gz</span>
<span class="na">checksums</span><span class="pi">:</span>
<span class="na">md5</span><span class="pi">:</span> <span class="s">2edd3a155dcbb6632029accd2926d33b</span>
<span class="na">sha1</span><span class="pi">:</span> <span class="s">776b13c7ff8e8b120a1ea4910a8b8b64c289e6b6</span>
<span class="na">extract</span><span class="pi">:</span> <span class="s">Yes</span> <span class="c1"># Whether an archive should be extracted</span>
<span class="na">icon</span><span class="pi">:</span> <span class="s">foo.png</span> <span class="c1"># An icon for GUI package managers</span>
<span class="na">licenses</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">MIT</span><span class="pi">]</span>
<span class="na">url</span><span class="pi">:</span> <span class="s">http://www.foo.org</span>
<span class="na">categories</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">fictional</span><span class="pi">,</span> <span class="nv">example</span><span class="pi">]</span> <span class="c1"># Keywords associated with this package</span>
<span class="na">distributions</span><span class="pi">:</span> <span class="no">null</span> <span class="c1"># Allows to install a group of packages as a single</span>
<span class="c1"># target</span>
<span class="na">requires</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">bar >= </span><span class="m">2.1</span>
<span class="pi">-</span> <span class="s">eggs == 1.1.2</span>
<span class="na">provides</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">example</span>
<span class="na">conflicts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">foobar</span>
<span class="na">optional</span><span class="pi">:</span> <span class="c1"># Keys denote optional packages, values describe why they are</span>
<span class="c1"># optional and what features they provide</span>
<span class="pi">-</span> <span class="na">eggs</span><span class="pi">:</span> <span class="s">To make a nice omelette</span>
<span class="pi">-</span> <span class="na">spam</span><span class="pi">:</span> <span class="s">If you like that sort of thing...</span>
<span class="na">extensions</span><span class="pi">:</span> <span class="c1"># User-defined data</span>
<span class="c1"># The following is metadata used by makepkg, but not used by other</span>
<span class="c1"># common package managers.</span>
<span class="na">options</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">trip</span><span class="pi">,</span> <span class="s2">"</span><span class="s">!docs"</span><span class="pi">,</span> <span class="nv">libtool</span><span class="pi">,</span> <span class="nv">emptydirs</span><span class="pi">,</span> <span class="nv">zipman</span><span class="pi">]</span>
<span class="na">backup</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/etc/foo.rc</span>
<span class="na">install</span><span class="pi">:</span> <span class="s">foo.install</span>
<span class="na">build</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">./configure --prefix=/usr</span>
<span class="s">make</span>
<span class="s">make DESTDIR=$pkgdir install</span>
<span class="c1">#...</span>
</code></pre></div></div>
<p>Most of these fields are analogous to those in the current PKGBUILD format.
The major difference is with <code class="language-plaintext highlighter-rouge">architectures</code>. The keys of this hash table
denote supported architectures. The <em>any</em> architecture has two sources. The
URI and checksums are defined. No longer is a conditional required. The
appropriate architecture is simple retrieved and its sources are used. The
source URLs and checksums now have a one-to-one mapping, reducing human
error. There is still one problem with this, however. If the sources do not
differ for multiple architectures, there will be duplication. To rectify this
to some degree, anchors can be used:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">architectures</span><span class="pi">:</span>
<span class="na">i686</span><span class="pi">:</span> <span class="nl">&common_sources</span>
<span class="na">sources</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uri</span><span class="pi">:</span> <span class="s">http://www.foo.org/-.tar.gz</span>
<span class="na">checksums</span><span class="pi">:</span>
<span class="na">md5</span><span class="pi">:</span> <span class="s">fd085a845298afb36f6676feac855e63</span>
<span class="na">sha1</span><span class="pi">:</span> <span class="s">e19f73b340aeae43b98908006094af62e0c7b5b9</span>
<span class="na">x86_64</span><span class="pi">:</span> <span class="nv">*common_sources</span>
</code></pre></div></div>
<p>In some cases it might even be useful to exploit hash merges to specify a
common subset of sources and add architecture-specific ones.</p>
<p>Another re-use of data was common in PKGBUILDs - using variables to reference
the package name and version in sources. It looked something like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">pkgname</span><span class="o">=</span><span class="s2">"foo"</span>
<span class="nv">pkgver</span><span class="o">=</span>1.0
<span class="nb">source</span><span class="o">=(</span><span class="s2">"http://foo.org/</span><span class="k">${</span><span class="nv">pkgname</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">pkgver</span><span class="k">}</span><span class="s2">.tar.gz"</span><span class="o">)</span>
</code></pre></div></div>
<p>You might have noticed that a similar syntax was used in the uri:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- uri: <span class="s2">"http://www.foo.org/</span><span class="k">${</span><span class="nv">name</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="nv">version</span><span class="k">}</span><span class="s2">.tar.gz"</span>
</code></pre></div></div>
<p>This is not something that YAML supports - it would have to be parsed
separately. I have not decided on this format yet, and perhaps YAML does
indeed have an appropriate feature. For now these values can either be
hardcoded, or parsed on the second pass of the data.</p>
<p>The <code class="language-plaintext highlighter-rouge">extensions</code> field is the interesting part. If the specification doesn’t
have some required data, such as the <code class="language-plaintext highlighter-rouge">options</code> field in the PKGBUILD
specification, it can be added here.</p>Sebastian NowickiOne of my favourite things about Unix-like operating systems is package management. I discuss the Archlinux PKGBUILD format and propose a new metadata format for packages.