SoftwareWeibian [index]

    Weibian is a software for taking scientific notes in the spirit of Forester, using Typst as the markup language.

    For the design thoughts behind Weibian, see posts in Weibian Blog .

    How to set up Weibian [0007]

    Weibian is implemented in Rust. To set up Weibian on your local machine, you can either install using cargo:

    cargo install --locked --git https://github.com/hanwenguo/weibian.git
    cargo install --locked --git https://github.com/hanwenguo/weibian.git

    Or download the prebuilt binary from the releases page.

    Weibian uses a simple project structure to organize your notes and resources. Say you have a directory called notes/ to store all your notes. Usually, you would want to have a structure like this:

    notes/
    ├── .wb/        # Weibian configuration and templates
    │   ├── config.toml  # Configuration file (optional)
    │   ├── templates/   # HTML templates
    ├── public/        # Resource files to be copied to output directory
    │   └── ...
    ├── dist/          # Default output directory
    │   └── ...
    └── typ/           # Note files in Typst format
        └── ...
    notes/
    ├── .wb/        # Weibian configuration and templates
    │   ├── config.toml  # Configuration file (optional)
    │   ├── templates/   # HTML templates
    ├── public/        # Resource files to be copied to output directory
    │   └── ...
    ├── dist/          # Default output directory
    │   └── ...
    └── typ/           # Note files in Typst format
        └── ...

    Most of the above directories is the default configuration, which can be overridden by passing command line arguments when running Weibian. The .wb directory is necessary for now, since it keeps the HTML template files and configuration file (if any). You must create the .wb directory manually for now; in the future, Weibian may provide a command to initialize a project structure automatically.

    Writing in Weibian [0008]

    Weibian works by exporting your notes written in Typst format to HTML using the HTML export feature of Typst, then post-processing the exported HTML files to create a website. Therefore, writing notes in Weibian is essentially writing Typst documents with some conventions. For a demonstration of these conventions, see the repository of Weibian itself.

    Rendering Process [rendering-process]

      This section describes the internal rendering process of Weibian. To quickly get started with writing in Weibian, you can skip this section for now and use the default templates in the typ directory in the repository of Weibian, which is also the template for this very site.

      Weibian will first export each note in the input directory to HTML. This does not involve any special processing; the Typst files are compiled as-is, and user is responsible for generating HTML files with the Weibian conventions described below. The generated HTML is not used for display directly nor saved as file, but rather as an in-memory intermediate representation for further processing.

      In the <body> of the HTML, there could be three special custom elements: <wb-transclusion target="wb:..." show-metadata="..." expanded="..." disable-numbering="..." demote-headings="..."></wb-transclusion>, <wb-internal-link target="wb:...">...</wb-internal-link> and <wb-cite target="wb:..."></wb-cite>. <wb-transclusion> is used to represent transcluded notes, <wb-internal-link> is used for internal links between notes, and <wb-cite> is used for citations to notes, which is basically a special kind of internal link. For <wb-transclusion>, its body must be empty; the target attribute, starting with wb:, specifies the ID (not including the wb: prefix) of the note to be transcluded, while show-metadata and expanded are boolean attributes that control the display of metadata and whether the transclusion is expanded by default, respectively; due to limitations in Typst’s HTML export capabilities, their values are represented as strings (“true” or “false”). disable-numbering is a boolean attribute that controls whether to disable heading numbering in the transcluded content, and demote-headings is a non-negative integer attribute that controls how many levels to demote headings in the transcluded content (e.g., if demote-headings is 1, then all h1 headings in the transcluded content will be demoted to h2, all h2 will be demoted to h3, and so on). For <wb-internal-link>, the target attribute specifies the ID of the note to link to, and its body contains the link text. For <wb-cite>, the target attribute specifies the ID of the note to cite, and its body contains the citation text.

      Then, Weibian extracts information from the generated HTML, and use the Tera templating engine and user-supplied templates to produce the final HTML files for the notes. By default, Weibian looks for templates in .wb/templates/.

      This rendering process begins by parsing the intermediate HTMLs to build a transclusion graph with respect to the <wb-transclusion> elements. Then, the notes are processed in topological order. For each note, the aforementioned custom elements are replaced with the actual content they represent.

      First, the transclusion and linking relationships are analyzed to build a transclusion graph. Each note is represented as a node in the graph, and a directed edge from node A to node B exists if note A transcludes note B. If there are cycles in the transclusion graph, Weibian will report an error and abort the rendering process, as cyclic transclusions are not supported.

      Then, transclusions are processed. For <wb-transclusion>, it is rendered via the transclusion.html template, which is provided with a a transclusion context (transclusion.target, transclusion.show_metadata, transclusion.expanded, transclusion.hide_numbering, transclusion.demote_headings, transclusion.metadata, transclusion.content). The transclusion.target, transclusion.show_metadata, transclusion.expanded, transclusion.hide_numbering, and transclusion.demote_headings are extracted from the corresponding attributes of the <wb-transclusion> element, while transclusion.metadata is the metadata of the target note, extracted from the <meta> tags in the <head> of the intermediate HTML of the target note, and transclusion.content is the processed content of the target note’s final HTML file, to help simplify transclusion rendering in templates. Two Tera filters are registered to help transclusion rendering: wb_hide_numbering and wb_demote_headings. They apply unconditionally; template conditionals decide whether to invoke them (see the default transclusion.html). The result of rendering this template replaces the corresponding <wb-transclusion> element in the final HTML file. By processing the notes in topological order, the target note should have already been processed when processing the current note. After this step, there should be only <wb-internal-link> and <wb-cite> elements left in the HTML file.

      For <wb-internal-link>, it is rendered via the internal_link.html template, which is provided with a link context (link.target, link.text, link.href). The link.target and link.text are extracted from the corresponding attribute and body of the <wb-internal-link> element, while link.href is the generated URL to the target note’s final HTML file, to help simplify link generation in templates. When the body of the <wb-internal-link> element is empty, the title of the target note is used as the link text. The result of rendering this template replaces the corresponding <wb-internal-link> element in the final HTML file. The rendering process for <wb-cite> is similar, except that it uses the citation.html template and a citation context (citation.target, citation.text, citation.href).

      Then, backmatters are generated for each note. As for now, Weibian supports four types of backmatter sections: contexts, references, backlinks, and related notes.

      • A context for note A is defined as any note that directly transcludes note A.
      • A reference from note A to note B exists if note A links to note B via an citation link.
      • A backlink from note A to note B exists if note B links to note A via an internal link.
      • A related note to note A to note B exists if note A links to note B via an internal link.

      The content of each backmatter section is generated by rendering transclusions of all notes relevant to that backmatter section with show-metadata="true", expanded="false", disable-numbering="true", and demote-headings="<number>" options. Then, for each backmatter section, a Tera context is created with title being the name of the backmatter section (e.g., “Backlinks”, “Contexts”) and content being the HTML of the rendered transclusion described above. These contexts are packed into an array and passed to the note.html template as note.backmatter_sections for rendering, see the next paragraph.

      Finally, the final HTML file for each note will be constructed. The template for that is note.html. It receives a note context (note.id, note.title, note.metadata, note.head, note.content, note.toc, note.backmatter). note.id and note.title are extracted from the corresponding <meta> tags, provided for convenience. The note.metadata is a map of metadata key-value pairs extracted from <meta> tags with name and content attributes in the <head> section of the intermediate HTML. The note.head is the raw HTML content of the <head> section of the intermediate HTML. note.content is the processed content of the <body> section of the intermediate HTML, with all transclusions and internal links resolved as described above. note.backmatter_sections is an array of backmatter section contexts as described in the previous paragraph. The toc field is an array of Heading objects, where each Heading object has the following structure:

      // The hX level
      level: 1 | 2 | 3 | 4 | 5 | 6;
      // The `id` attribute of the heading tag
      id: String;
      // The inner HTML of the heading tag
      content: String;
      // Whether the heading has the "disable-numbering" class
      disable_numbering: Bool;
      // All lower level headers below this header
      children: Array<Heading>;
      // The hX level
      level: 1 | 2 | 3 | 4 | 5 | 6;
      // The `id` attribute of the heading tag
      id: String;
      // The inner HTML of the heading tag
      content: String;
      // Whether the heading has the "disable-numbering" class
      disable_numbering: Bool;
      // All lower level headers below this header
      children: Array<Heading>;

      Along all the rendering process, a site context (site.root_dir, site.trailing_slash, site.domain) is also provided to all templates to help with link generation and other site-wide settings.

      When the document has export-pdf metadata set to true, an additional PDF export will be generated for the note. The PDF is simply generated by running the Typst compiler on the original Typst file, with an extra wb-id-filename-map-file input set to the root-absolute path of a JSON file that maps note IDs to their corresponding source Typst file root-absolute paths, to help resolving internal links and transclusions in the PDF export. The PDF file is saved with the name <identifier>.pdf in the pdf/ subdirectory of the output directory. The PDF export is independent from the HTML export; it does not use the intermediate HTML nor the Tera templates, and it is triggered solely by the presence of the export-pdf metadata field in the original Typst file, which is processed in the default template to add a corresponding <meta name="export-pdf" content="true"> tag in the intermediate HTML. For a starter template for both HTML and PDF export, see the source of this very site.

      Using Configuration File [using-configuration-file]

        Weibian supports a .weibian/config.toml configuration file to allow users to set project options such as input/output directories, public assets directory, and other preferences. CLI flags override config values.

        The following is an example configuration file:

        [files]
        input_dir = "typ"
        output_dir = "dist"
        public_dir = "public"
        # include = ["**/*.typ"]  # optional; defaults to all files in input_dir
        # exclude = ["draft-*"]   # optional; exclude has priority over include
        # the above is the equivalent of the corresponding CLI flags

        [site]
        domain = "example.com" # the domain of the site; used for generating absolute URLs
        root_dir = "/" # the root directory of the site; for example, if the site is hosted at example.com/notes/, set root_dir = "/notes/"
        trailing_slash = true # if true, the final URL of each note will have a trailing slash
        [files]
        input_dir = "typ"
        output_dir = "dist"
        public_dir = "public"
        # include = ["**/*.typ"]  # optional; defaults to all files in input_dir
        # exclude = ["draft-*"]   # optional; exclude has priority over include
        # the above is the equivalent of the corresponding CLI flags

        [site]
        domain = "example.com" # the domain of the site; used for generating absolute URLs
        root_dir = "/" # the root directory of the site; for example, if the site is hosted at example.com/notes/, set root_dir = "/notes/"
        trailing_slash = true # if true, the final URL of each note will have a trailing slash

        The configuration file is parsed at the start of the program, and values are used as defaults for the corresponding CLI flags. If cache_dir is omitted, Weibian uses a project-specific directory under the system temporary directory for intermediate HTML. By default, the configuration file is looked for in .wb/config.toml relative to the project root, but a different path can be specified with --config-file <PATH>. Site settings can also be overridden via CLI with --site-domain, --site-root-dir, and --trailing-slash <BOOL>. The settings in the [site] section are also passed to the Typst compiler as inputs, with the prefix wb- and underscores converted to hyphens (e.g., site.domain becomes wb-domain).

        The trailing_slash option will affect how internal links are generated and how the output files are organized. If trailing_slash is true, each note will be saved in a subdirectory named after its ID, with an index.html file inside (e.g., a note with ID note-123 will be saved as dist/note-123/index.html). If false, each note will be saved directly as an HTML file named after its ID (e.g., dist/note-123.html). The root_dir setting only affects link generation; it does not change where files are written. Special case: a note with ID index is always saved as dist/index.html and links to the site root.

        Use Emacs denote package to write in Weibian [0009]

        If you use Emacs, Weibian is accompanied by an Emacs Lisp package providing the integration of Weibian and the Denote package. The following is an example of configuration. However, since everyone has different templates, there’s a lot of variables to tweak, and you need to read the source code of the package (it’s not very big though) to understand how to customize it. A rewrite of the package to make it more idiomatic is planned.

        (use-package denote
          :bind (("C-c n n" . denote)) ;; add your keybindings
          :config
          (setq denote-directory (expand-file-name "~/Documents/notes/typ/"))
          (setq denote-prompts '(signature title))
          (setq denote-excluded-directories-regexp "_template")

          ;;; Use incrementing base-36 numbers as id
          ;;; This function assumes that IDENTIFIERS is a list of base-36 strings
          ;;; i.e. 4-character strings consisting of numbers and uppercase letters
          (defun my/denote-get-next-base36 (identifiers)
            (let ((maxs nil))
              (dolist (s identifiers)
                (let ((u (upcase s)))
                  (when (or (null maxs) (string> u maxs))
                    (setq maxs u))))
              ;; increment maxs
              (let ((buf (copy-sequence maxs))
                    (i 3)
                    (carry 1))
                (while (and (>= i 0) (= carry 1))
                  (let* ((d (aref buf i)))
                    (if (= d ?Z)
                        (progn
                          (aset buf i ?0)
                          (setq carry 1))
                      (if (= d ?9)
                          (aset buf i ?A)
                        (aset buf i (+ d 1)))
                      (setq carry 0)))
                  (setq i (1- i)))
                buf)))

          (defun my/denote-generate-base36-identifier (initial-identifier _date)
            (let ((denote-used-identifiers (or denote-used-identifiers (denote--get-all-used-ids))))
              (cond (;; Always use the supplied initial-identifier if possible,
                     ;; regardless of format.
                     (and initial-identifier
                          (not (gethash initial-identifier denote-used-identifiers)))
                     initial-identifier)
                    (;; Else, the supplied initial-identifier is nil or it is already
                     ;; used. Ignore it and generate a valid identifier with the right
                     ;; format.
                     t
                     (let* ((identifiers (hash-table-keys denote-used-identifiers))
                            (case-fold-search nil)
                            (base36-identifiers (seq-filter (lambda (id) (string-match-p "[0-9A-Z]\\{4\\}" id)) identifiers)))
                       (if base36-identifiers
                           (my/denote-get-next-base36 base36-identifiers)
                         "0000"))))))
          (setq denote-get-identifier-function #'my/denote-generate-base36-identifier))

        (use-package typst-ts-mode)

        (use-package denote-weibian
          :load-path "/path/to/weibian/directory/"
          :after (denote)
          :demand t
          :bind (("C-c n b" . denote-weibian-backlinks)
                 ("C-c n c" . denote-weibian-contexts)
                 ("C-c n t" . denote-weibian-transclude))
          :config
          (push denote-weibian-file-type denote-file-types)

          (setq denote-file-name-slug-functions
                '((title . denote-sluggify-title)
                  (signature . identity)
                  (keyword . denote-sluggify-keyword))))
        (use-package denote
          :bind (("C-c n n" . denote)) ;; add your keybindings
          :config
          (setq denote-directory (expand-file-name "~/Documents/notes/typ/"))
          (setq denote-prompts '(signature title))
          (setq denote-excluded-directories-regexp "_template")

          ;;; Use incrementing base-36 numbers as id
          ;;; This function assumes that IDENTIFIERS is a list of base-36 strings
          ;;; i.e. 4-character strings consisting of numbers and uppercase letters
          (defun my/denote-get-next-base36 (identifiers)
            (let ((maxs nil))
              (dolist (s identifiers)
                (let ((u (upcase s)))
                  (when (or (null maxs) (string> u maxs))
                    (setq maxs u))))
              ;; increment maxs
              (let ((buf (copy-sequence maxs))
                    (i 3)
                    (carry 1))
                (while (and (>= i 0) (= carry 1))
                  (let* ((d (aref buf i)))
                    (if (= d ?Z)
                        (progn
                          (aset buf i ?0)
                          (setq carry 1))
                      (if (= d ?9)
                          (aset buf i ?A)
                        (aset buf i (+ d 1)))
                      (setq carry 0)))
                  (setq i (1- i)))
                buf)))

          (defun my/denote-generate-base36-identifier (initial-identifier _date)
            (let ((denote-used-identifiers (or denote-used-identifiers (denote--get-all-used-ids))))
              (cond (;; Always use the supplied initial-identifier if possible,
                     ;; regardless of format.
                     (and initial-identifier
                          (not (gethash initial-identifier denote-used-identifiers)))
                     initial-identifier)
                    (;; Else, the supplied initial-identifier is nil or it is already
                     ;; used. Ignore it and generate a valid identifier with the right
                     ;; format.
                     t
                     (let* ((identifiers (hash-table-keys denote-used-identifiers))
                            (case-fold-search nil)
                            (base36-identifiers (seq-filter (lambda (id) (string-match-p "[0-9A-Z]\\{4\\}" id)) identifiers)))
                       (if base36-identifiers
                           (my/denote-get-next-base36 base36-identifiers)
                         "0000"))))))
          (setq denote-get-identifier-function #'my/denote-generate-base36-identifier))

        (use-package typst-ts-mode)

        (use-package denote-weibian
          :load-path "/path/to/weibian/directory/"
          :after (denote)
          :demand t
          :bind (("C-c n b" . denote-weibian-backlinks)
                 ("C-c n c" . denote-weibian-contexts)
                 ("C-c n t" . denote-weibian-transclude))
          :config
          (push denote-weibian-file-type denote-file-types)

          (setq denote-file-name-slug-functions
                '((title . denote-sluggify-title)
                  (signature . identity)
                  (keyword . denote-sluggify-keyword))))