@bnjbvr - rust

Publish a Zola blog with Gitlab CI, real fast

2024-11-02T00:00:00+00:00

The two or three regular visitors of this blog (hi friends!) might have noticed I’ve changed the design, as of this year. I’ve actually moved from using the Pelican</a> static blog generator to using Zola</a>. This article shows the continuous integration I’ve set up to automatically build and push content to my server, Real Fast™.

Was there anything wrong with Pelican</code>? Not really. It has served me well over the years, and its naming coming from an anagram of calepin</code>, French for a “small notebook”, is still just genius. Installing on a new machine was a bit of a pain as I needed to recall which Python / virtualenv / etc. commands were required to install dependencies, but over the years I’ve made a Makefile to simplify that. And it didn’t happen that often though; at most I’d run such a command once every two years or so (which is, arguably, already a lot if that’s the frequency at which one switches machines).

I mostly moved to Zola because I’m a zealot / member of the Rust evangelism strike force</a> on my spare time, and want to support tools created with Rust, which are usually blazing 😎 fast 🚀 1</a>, in addition to being super safe. And, having a statically linked binary is super convenient, and notably super nice for continuous integration and deployment purposes. Indeed, it’s trivial to cache a single binary in CI, and not have to worry about installing/caching dependencies and so on. In addition to that, Zola supports front-matter annotations with TOML or YAML, so it offered a nice migration path from Pelican, who uses YAML for frontmatters.

I’ve been using Gitlab</a>, notably the Framagit</a> instance hosted by the good fellows at Framasoft</a> (give them money to support them!), for the repository hosting the sources of my blog. As such, I’ve wanted to be able to push to the repository, and have the CI build and publish to my website.

Now, let me explain how I’ve did it. These instructions are valid for GitLab Community Edition v17.5.1 ; some things may change in newer versions, so I can’t guarantee they’ll work forever. There’s a first stage that will build the public website using a cached Zola binary, if possible, or grab it from the Github releases website otherwise. The second stage2</a> will upload it to the server, using rsync</code>. I’d recommend creating a new user just for this task, with limited SSH access to a single directory, that is, where the generated HTML will live. With caching, each stage takes at most 10 seconds to run, which I find… acceptable 😁.

Here’s the .gitlab-ci.yml</code> file I’ve checked in, heavily commented for your (and my future self’s) best understanding:

default</span>:</span></span>
  image</span>:</span> debian:stable-slim</span></span>
</span>
variables</span>:</span></span>
  # The runner will be able to pull your Zola theme when the strategy is</span></span>
  # set to "recursive".</span></span>
  GIT_SUBMODULE_STRATEGY</span>: "</span>recursive</span>"</span></span>
</span>
  # If you don't set a version here, your site will be built with the latest</span></span>
  # version of Zola available in GitHub releases.</span></span>
  # Use the semver (x.y.z) format to specify a version. For example: "0.17.2" or "0.18.0".</span></span>
  ZOLA_VERSION</span>:</span></span>
    description</span>: "</span>The version of Zola used to build the site.</span>"</span></span>
    value</span>: "</span>0.19.1</span>"</span></span>
</span>
build</span>:</span></span>
  stage</span>:</span> build</span></span>
</span>
  # Cache the Zola binary based on its version, to avoid conflicts between different versions.</span></span>
  cache</span>:</span></span>
    key</span>:</span> $ZOLA_VERSION</span></span>
    paths</span>:</span></span>
     # $CI_PROJECT_DIR is the current working directory in subsequent steps.</span></span>
      -</span> "</span>$CI_PROJECT_DIR/zola</span>"</span></span>
</span>
  script</span>:</span></span>
    -</span> |</span></span>
      if [ ! -e "$CI_PROJECT_DIR/zola" ]; then</span></span>
        echo "Downloading Zola…"</span></span>
</span>
        # Download enough to use `wget`.</span></span>
        apt-get update --assume-yes && apt-get install --assume-yes --no-install-recommends wget ca-certificates</span></span>
</span>
        if [ $ZOLA_VERSION ]; then</span></span>
          zola_url="https://github.com/getzola/zola/releases/download/v$ZOLA_VERSION/zola-v$ZOLA_VERSION-x86_64-unknown-linux-gnu.tar.gz"</span></span>
          if ! wget --quiet --spider $zola_url; then</span></span>
            echo "A Zola release with the specified version could not be found.";</span></span>
            exit 1;</span></span>
          fi</span></span>
        else</span></span>
          github_api_url="https://api.github.com/repos/getzola/zola/releases/latest"</span></span>
          zola_url=$(</span></span>
            wget --output-document - $github_api_url |</span></span>
            grep "browser_download_url.*linux-gnu.tar.gz" |</span></span>
            cut --delimiter : --fields 2,3 |</span></span>
            tr --delete "\" "</span></span>
          )</span></span>
        fi</span></span>
</span>
        wget $zola_url</span></span>
        tar -xzf *.tar.gz</span></span>
      else</span></span>
        echo "Reusing cached Zola…"</span></span>
      fi</span></span>
    -</span> |</span></span>
      $CI_PROJECT_DIR/zola build</span></span>
</span>
  # The built artifacts will be put in the `public/` directory, and reused during the next stage.</span></span>
  artifacts</span>:</span></span>
    paths</span>:</span></span>
      -</span> public/</span></span>
    expire_in</span>:</span> 1 day</span></span>
</span>
deploy</span>:</span></span>
  stage</span>:</span> deploy</span></span>
  only</span>:</span></span>
  -</span> main</span></span>
  dependencies</span>:</span></span>
  -</span> build</span></span>
  script</span>:</span></span>
  # Install rsync and ssh, if needs be.</span></span>
  -</span> apt-get update -qq && apt-get install -y -qq rsync</span></span>
  -</span> '</span>which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )</span>'</span></span>
  -</span> eval $(ssh-agent -s)</span></span>
  # Set SSH private key, and define the right permissions to not trigger security errors.</span></span>
  -</span> echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null</span></span>
  -</span> mkdir -p ~/.ssh</span></span>
  -</span> chmod 700 ~/.ssh</span></span>
  # Set SSH known hosts, and define the right permissions to not trigger security errors.</span></span>
  -</span> echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts</span></span>
  -</span> chmod 644 ~/.ssh/known_hosts</span></span>
  # Run rsync.</span></span>
  # Note: -avzh = append / verbose / compress / human-readable</span></span>
  -</span> rsync -avzh --delete public/* -e "ssh -p $SSH_PORT" $SSH_USERNAME@$SSH_HOST:$SSH_TARGET_DIR</span></span></code></pre>
After you’ve set this up, you need to fill some secrets on your CI, by going to Settings</code>,
then CI/CD</code>, then fill all the following variables under Variables</code>. Make sure to create them
as Masked</code> variables, if not Masked and hidden</code>, unless you’d like the SSH private key to
leak 🤡.</p>

SSH_HOST</code>: server’s host, e.g. mysupersecretserver.bouvier.cc</code></li>
SSH_PORT</code>: the port to be used for connecting with SSH, e.g. 22 by default.</li>
SSH_KNOWN_HOSTS</code>: a copy of a simplified ~/.ssh/known_hosts</code> file, stripped down to only
contain lines related to the above SSH_HOST</code>. This is to avoid having the CI task to confirm “Are
you sure you want to trust this host” when connecting for the first time to the server.</li>
SSH_TARGET_DIR</code>: the final directory where the generated HTML should live, e.g. /var/www/myblog</code>.</li>
SSH_USERNAME</code>: the username of the (Unix) user used in the CI task to upload generated artifacts
to your Web server…</li>
SSH_PRIVATE_KEY</code>: …and its private key.</li>
</ul>
Hope this was useful!</p>



Don’t pay attention to the emojis, they’re a meme at this point. ↩</a></p>
</li>

Using two stages is inherited from my previous setup using Pelican, and might be overkill
since the each stage takes at most 10 seconds to run, with a hot cache. ↩</a></p>
</li>
</ol>
</section>

cargo-machete, find unused dependencies quickly 2022-04-27T23:17:23+00:00 cargo-machete</code> is a new Cargo tool that detects unused dependencies in Rust projects, in a fast (yet imprecise) way. As of today you can install it with cargo install cargo-machete</code> and then run it with cargo machete</code> from any folder that contains a workspace or crate, to find if you have potentially unused dependencies. Beware, it can report a few false positives! 🔗</a>Problem statement</h2> When developers hack on code, it’s a pretty common to reuse software that already exists and has been written, optimized, and battle-tested by many others. In fact, that’s a core idea of the open-source movement, and one historical reason for its existence. When zooming in into the Rust programming language case, my opinion is that it is also a key reason why Rust has been so successful: having plenty of crates doing everything you might need, already implemented for you and at hand’s reach on crates.io</code>. Plus, having the wonder of a one-does-it-all Cargo tool that makes it very easy to use those crates as dependencies in your project. 1</a> However, this comes with a price: sometimes you add a dependency because it’s useful at a particular point in time. Much later, it’s not useful anymore, but you may have forgotten about it. And then, the dependency remains as a zombie in your Cargo.toml</code> file. Cargo will include it in the compilation graph, despite the compilation artifacts not being used at all. The unused dependency will just stay there, silently weep, waiting for you to recall it exists. Of course, the problem can even become worse: maybe you maintain several crates that have unused dependencies. Or maybe you work with many crates as part of a workspace, and each may have unused dependencies. Or simply you use many dependencies yourself, and some may include unused dependencies. If you’ve published your crates and others use those, then everyone could also compile unused dependencies. At the scale of the entire Rust crates ecosystem, it can have a huge impact on the compile times, produced heat and wasted energy. 🔗</a>Have you heard about our lord and savior, cargo-udeps</code></h2> There’s already a nice tool for this in the ecosystem: cargo-udeps</code></a>. It will compile your crate (or workspace) and then infer from the compiled artifacts what dependencies are used by your project, and thus show you which dependencies are unused. That’s great, but the way it works forces a few tradeoffs: it requires to compile the whole crate with the Rustc nightly</code> compiler. For me that means recompiling the whole project from scratch, most of the time, since I’m mostly using stable rustc as my daily driver.</li> if you compile for multiple targets (i.e. different combinations of CPU flavor, OS, environment, etc.), you’d need to run cargo-udeps</code> on each of those to find per-target unused dependencies. For instance, if a dependency is only configured when compiling for x86_64 machines, then it may be flagged as unused on every other configuration.</li> most of all, since it look at compilation artifacts, it cannot know if a specific dependency is directly used by your crate, or indirectly, leading to somehwat mystifying results in the case of workspaces.</li> </ul> Let’s dive a bit deeper into the last item, which I’ll refer to as the transitively-used dependencies problem. Say you have your project AAA</code> that contains a dependency to serde</code> in its Cargo.toml</code> file, while it’s not directly used by your code. In fact, if you did a text-search of serde</code> in AAA</code>’s code with grep</code>, you wouldn’t find a single match2</a>. But now AAA</code> is using another crate, AB</code>, that itself depends on serde</code>. cargo-udeps</code> will see that serde</code> is used overall, so it cannot let you know that AAA</code>’s Cargo.toml</code> file references an unused dependency to serde</code>. How is this a problem? After all, if the workspace uses serde</code> even indirectly, then we will have to compile it at some point, so it’s not like it’s really unused. First of all, the AAA</code> crate might be using a different version of serde</code> than the AB</code> crate, and this could result in different copies of the same crate in your workspace. Note there are other nice tools that automatically detect this kind of situation (hi there cargo-deny</a>). Second, the order in which crates are compiled has an impact on compilation parallelism, and having unused dependencies may add spurious synchronization points in the compilation graph. When a Rust crate gets compiled by Cargo, Cargo proceeds in two phases: first, it collects information so as to unlock the compilation of other crates further down the road that may depend on this particular one. I don’t know precisely what it entails, but one can make educated guesses: parse the code, analyze which items are pub</code>lic, compute memory layouts for public types, collect type information and so on and so forth.</li> then, it does the actual compilation: optimize and generate the actual machine code for that particular crate, that will be later linked with other artifacts to form the final executable program.</li> </ul> The advantage of this two-phases scheme is that once Cargo is done with phase 1 for a particular crate, it can kick off the same process for other crates up the dependency tree, while it runs phase 2 concurrently. With a multi-core machine as is the norm on desktop computers, it’s almost certain that this will bring speedups! For instance, consider the following Cargo.toml</code> file from our previous example project: [dependencies] serde = "1.0"</code></pre> Then a possible compilation graph could look like that: In this case, ab</code> phase 1 can start as soon as serde</code> phase 1 has finished, while serde</code>’s compilation phase 2 happens in the background. If you’re interested in reducing the overall compile times of your Rust project, I would strongly suggest to go read Rust’s documentation around timings visualization</a>. Crates which spend lots of time in the first phase (or more generally, in both phases) are basically pipelining bottlenecks, so identifying/removing/working around them overall speeds up compile times. Back to our small unused dependency problem: an unused dependency in your Cargo.toml</code> may block the compilation of other crates up the dependency tree, and thus may slow down the whole compilation process by creating useless check points. Consider a crate C</code> that depends on crates A</code> and B</code>, with B</code> actually unused: Here, the compilation of the crate C</code> could start way earlier, but it’s blocking waiting for the compilation of B</code> to finish first, while it’s not even used! 🔗</a>Solving this, the naive way</h2> So when I was trying to confirm whether crates found by cargo-udeps</code> were actually used or not in my Rust projects, the thing I’d do would be to grep</code> (or better, use the blazingly fast Rust replacement ripgrep</a>) the crate’s name in the project. After all, the crate’s name is in the source directory, if and only if the crate is used, right? The answer is… mostly, yes. If we exclude dynamic code loading via mechanisms like dlopen</code> or WebAssembly, then there aren’t so many ways to use other crates directly, in Rust code. In fact, we can exhaustively enumerate all the syntax items to use other dependencies in Rust: use my_crate; use your_crate as my_crate; use { your_crate as my_crate }; extern crate my_crate; fn main() { my_crate::something(); }</code></pre> I’ve looked at a bit of Rust code now, and I haven’t seen other direct forms; if I am missing any, please let me know! Now, these are the most frequent ways to use a dependency, but there are in fact other ways: build.rs</code> scripts can generate code that could use other crates, and that would not be visible through a text search in the src/</code> directory, as the generated code is somewhere inside the target/build/</code> directory.</li> macros (procedural or not) can expand to code that’s using other crates, while the source code doesn’t explicitly mention them. For instance, the log_once</code></a> crate uses the log</code> macros in its own macros, but log_once</code> doesn’t depend on log</code> explicitly. It’s a bold and smart move: it breaks the coupling with the specific version of log</code> , and as long as the high-level API of log</code> is stable (which is the case), then log_once</code> works with any version of log</code>.</li> </ul> And then, there’s still a bit of room for some false positives: raw text submatches: e.g. if a crate is named bar</code>, then foobar::</code> would be a match if we’re doing a raw grep</code> search</li> text search isn’t syntaxic analysis, and we wouldn’t know if a match is in a comment (// use foo;</code>), or a string (String::from("use foo;")</code>).</li> </ul> But that would do most of the job, wouldn’t it? In particular, compared to cargo-udeps</code>, this approach doesn’t suffer from the transitively-used dependencies problem. If you look for a crate’s name in the src/</code> directory and it’s not there, it’s likely not used by your crate. The End. 🔗</a>A tedious process calls for automation, so I made a tool</h2> And I’ve called it cargo-machete</code>. Like a machete, it is very useful for quickly weeding out things, but it is very imprecise and you wouldn’t trust it at 100%. The gist of it is: find directories that might contain Rust projects, as indicated by the presence of a Cargo.toml</code> file</li> for each dependency, create an absolutely ugly regular expression that matches any of the syntaxic forms presented above. The regular expression does better than just raw text search, in particular it doesn’t run into the text submatch issue. then for each file in the project, try to match the regular expression against each line of any source file, and stop at the first successful match (which means the dependency is used)</li> </ul> </li> </ul> This tool is fast, because it combines the core library behind ripgrep</code> for matching regular expressions, with rayon for running it in parallel across all the dependencies of a project. On my machine, the problem is CPU-bound, because of the execution of the regular expression (and maybe thanks to my NVME storage too). That’s only one data point, but on this particular beefy desktop I use, it scans the entirety of the rust-lang/rust</code> repository in 1.08 seconds, or all of BytecodeAlliance/wasmtime</code> in 0.58 seconds. The tool is open source</a>, of course. As is the tradition for Cargo tools, it can be installed with: cargo install cargo-machete</code></pre> and then can be used, from any directory that contains Rust code (be it a workspace, a single project, or a directory on top of many Rust projects), with the following line: cargo machete</code></pre> Here’s an output example: > cargo machete Looking for crates in this directory and analyzing their dependencies... /home/ben/code/cargo-machete/integration-tests/with-bench/Cargo.toml -- no package, must be a workspace just-unused -- /home/ben/code/cargo-machete/integration-tests/just-unused/Cargo.toml: log unused-transitive -- /home/ben/code/cargo-machete/integration-tests/unused-transitive/Cargo.toml: lib1 Done!</code></pre> There are false positives: code generated via macros or build scripts aren’t inspected as they’re not in the src/</code> directory and cargo-machete</code> doesn’t run any compile step. For instance, if a project depends on log</code> , but uses it only through log_once</code>, then cargo-machete</code> will incorrectly flag log</code> as an unused dependency. The good news is that, thanks to a contribution from @daniel5151</code>, you can specify known false positives in the Cargo.toml</code> file of your crate, allowing use of cargo-machete</code> in CI setups: [package.metadata.cargo-machete] ignored = ["log"] # false positive, used by log_once! macro</code></pre> As far as I know, the risk for false negatives (i.e. crates that are unused, but the tool thinks they’re used) is pretty low. One such instance would be a multi-line string containing one of the use</code> forms, but that seems rather unlikely to be present in most Rust projects. The tool is still a bit rough, but it’s been already quite useful for some projects I’ve been working on! In a particular work project, most unused dependencies were transitively used and compiled, but the rejiggering of the compilation graph lead to a 5% compile time speedup overall. Good impact over effort ratio. 🔗</a>What about other languages?</h2> What makes this possible in Rust, and could it be extended to other languages? Dynamic languages by nature dynamically load code, but there are still ways to try to automate detecting unused dependencies same as cargo-machete</code> does. Consider JavaScript and its require</code> function, that can dynamically evaluate a string that’s a path to a file with code we want to import. Since there’s an infinity of ways to create a string, we can’t just perfectly rely on finding require("abc")</code> and assume that if not present, then abc</code> isn’t used. Ditto with import</code> statements, which can evaluate dynamic sources. That being said, if JS code is restricted to use require</code> statements with only static strings or static import</code> statements, then this may work too! Although when restricting to static require</code>s, even just loading the code in NodeJS would be sufficient to find unused dependencies with perfect accuraccy. Back to static languages, where I constrain the problem to non-dynamic dependencies (loaded via dlopen</code> etc.). In a language like C or C++, there are no unified module systems or package description (yet! although cmake</code> might be a de-facto standard). We can still apply this to header files, and look for their inclusion via #include</code> statements. Macros and preprocessed code would also throw a wrench in the process. Then some human intervention would still be required to eliminate the .c files, but I haven’t thought about it too much. Static analysis of compiled binaries might be simpler, for that matter. If we consider the problem for WebAssembly, we can frame it as “which imported functions are not used in the module”, potentially eliminating an entire range of host functions. In the simplest case, we could just look at the code section, through the function bodies, and see if there’s any reference to indices of every single imported function in call</code> opcodes. Then, there can be function Table</code>s referencing those, so we have to make sure no table elements reference the function. And if any table is mutable and publicly exposed via an export, then a user of the wasm module may reference any function declared in the wasm module, including imported functions, so all bets are off. Note dead-code elimination in wasm would be pretty similar and suffer from the same limitations: after all, a function dependency is just another kind of function, in wasm! Each format may have such idiosyncrasies like that. Static analysis of final binaries (as opposed to libraries) might be possible and reliable, though. 🔗</a>Closing thoughts</h2> For the sake of completeness, I should mention the existence of a rustc crate-wide lint for this, since May 2020 or so</a>: #![warn(unused_crate_dependencies)]</code>. This tells about unused crate dependencies directly as a Rust warning, which in my opinion would be the ideal end goal! Unfortunately, some Github comments suggest it suffers from having too many false positives, and still it requires compiling the code. @est31</code>, of cargo-udeps</code>’s fame, has been working on a better solution</a>. It seems to not be so far from completion, so between this and the Rust lint, I’m hopeful that there could be a time where we have a solution that is perfectly precise, with neither false positives nor false negatives. In the meanwhile, I hope that cargo-machete</code> can be useful to some of you, or that it inspires others to make similar quick-and-dirty tools, in Rust or in other languages. Thanks for reading this far, and please get in touch</a> if you have any thoughts about this! If you don’t know about the cargo-edit</code> tool that allows you to add a dependency in one line with cargo add serde</code> to your project: now you do</a>. ↩</a> </li> Wait for it… ↩</a> </li> </ol> </section> A primer on code generation in Cranelift 2021-02-17T19:00:42+00:00