How it Works
Supported Platforms
Usage
Configuration
- Annotated Sample Configuration
- output
- proxy
- file_urls
- feed.title
- feed.filename
- feed.config.url
- feed.config.item
- feed.config.heading
- feed.config.link
- feed.config.summary
- feed.config.date
- feed.config.media
Hosting, Updating, and Subscribing
Logging
Caveats & Error Handling
Caching

Documentation

How it Works

rsspls fetches each page specified by the configuration and extracts elements from the page using CSS selectors. For example elements are matched to determine the title and content of the feed entry. The generated feeds are written to an output directory. HTTP caching is used to only update the feed when the source page changes.

Supported Platforms

rsspls should work on all platforms supported by the Rust compiler including Linux, macOS, Windows, and BSD. Pre-compiled binaries are available for common platforms. See the install page for details.

Usage

rsspls [OPTIONS] -o OUTPUT_DIR

OPTIONS:
    -h, --help
            Prints this help information

    -c, --config
            Specify the path to the configuration file.
            $XDG_CONFIG_HOME/rsspls/feeds.toml is used if not supplied.

    -o, --output
            Directory to write generated feeds to.

    -V, --version
            Prints version information

FILES:
     ~/$XDG_CONFIG_HOME/rsspls/feeds.toml    rsspls configuration file.
     ~/$XDG_CONFIG_HOME/rsspls               Configuration directory.
     ~/XDG_CACHE_HOME/rsspls                 Cache directory.

     Note: XDG_CONFIG_HOME defaults to ~/.config, XDG_CACHE_HOME
     defaults to ~/.cache.

Configuration

Unless specified via the --config command line option rsspls reads its configuration from one of the following paths:

The configuration file is in TOML format.

The parts of the page to extract for the feed are specified using CSS selectors.

Annotated Sample Configuration

The sample file below demonstrates all the parts of the configuration.

# The configuration must start with the [rsspls] section
[rsspls]
# Optional output directory to write the feeds to. If not specified it must be supplied via
# the --output command line option.
output = "/tmp"
# Optional proxy address. If specified, all requests will be routed through it.
# The address needs to be in the format: protocol://ip_address:port
# The supported protocols are: http, https, socks and socks5h.
# It can also be specified as environment variable `http_proxy` or `HTTPS_PROXY`.
# The config file takes precedence, then the env vars in the above order.
# proxy = socks5://10.64.0.1:1080
# Optionally enable reading web pages from local files though file:// URLs.
# file_urls = false

# Next is the array of feeds, each one starts with [[feed]]
[[feed]]
# The title of the channel in the feed
title = "My Great RSS Feed"

# The output filename without the output directory to write this feed to.
# Note: this is a filename only, not a path. It should not contain slashes.
filename = "wezm.rss"

# Optional User-Agent header to be set for the HTTP request.
# user_agent = "Mozilla/5.0"

# The configuration for the feed
[feed.config]
# The URL of the web page to generate the feed from.
url = "https://www.wezm.net/"

# A CSS selector to select elements on the page that represent items in the feed.
item = "article"

# A CSS selector relative to `item` to an element that will supply the title for the item.
heading = "h3"

# A CSS selector relative to `item` to an element that will supply the link for the item.
# Note: This element must have a `href` attribute.
# Note: If not supplied rsspls will attempt to use the heading selector for link for backwards
#       compatibility with earlier versions. A message will be emitted in this case.
link = "h3 a"

# Optional CSS selector relative to `item` that will supply the content of the RSS item.
summary = ".post-body"

# Optional CSS selector relative to `item` that supplies media content (audio, video, image)
# to be added as an RSS enclosure.
# Note: The media URL must be given by the `src` or `href` attribute of the selected element.
# Note: Currently if the item does not match the media selector then it will be skipped.
# media = "figure img"

# Optional CSS selector relative to `item` that supples the publication date of the RSS item.
date = "time"

# Alternatively for more control `date` can be specified as a table:
# [feed.config.date]
# selector = "time"
# # Optional type of value being parsed.
# # Defaults to DateTime, can also be Date if you're parsing a value without a time.
# type = "Date" 
# # format of the date to parse. See the following for the syntax
# # https://time-rs.github.io/book/api/format-description.html
# format = "[day padding:none]/[month padding:none]/[year]" # will parse 1/2/1934 style dates

# A second example feed
[[feed]]
title = "Example Site"
filename = "example.rss"

[feed.config]
url = "https://example.com/"
item = "div"
heading = "a"

The first example above (for my blog WezM.net) matches HTML that looks like this:

<section class="posts-section">
  <h2>Recent Posts</h2>

  <article id="garage-door-monitor">
    <h3><a href="https://www.wezm.net/v2/posts/2022/garage-door-monitor/">Monitoring My Garage Door With a Raspberry Pi, Rust, and a 13Mb Linux System</a></h3>
    <div class="post-metadata">
      <div class="date-published">
        <time datetime="2022-04-20T06:38:27+10:00">20 April 2022</time>
      </div>
    </div>

    <div class="post-body">
      <p>I’ve accidentally left our garage door open a few times. To combat this I built
        a monitor that sends an alert via Mattermost when the door has been left open
        for more than 5 minutes. This turned out to be a super fun project. I used
        parts on hand as much as possible, implemented the monitoring application in
        Rust, and then built a stripped down Linux image to run it.
      </p>
    </div>

    <a href="https://www.wezm.net/v2/posts/2022/garage-door-monitor/">Continue Reading →</a>
  </article>

  <article id="monospace-kobo-ereader">
    <!-- another article -->
  </article>

  <!-- more articles -->

  <a href="https://www.wezm.net/v2/posts/">View more posts →</a>
</section>

output

Optional output directory to write the feeds to. If not specified it must be supplied via the --output command line option. Directory will be created if it does not exist.

Tilde expansion is performed on the path in the config file. This allows you to refer to the home directory of the user running rsspls. For example, ~/Documents/rsspls could be used to place the output in your Documents folder.

proxy

Optional proxy address. If specified, all requests will be routed through it. The address needs to be in the format: protocol://ip_address:port The supported protocols are: http, https, socks and socks5h.

The proxy for http and https requests can also be specified with the environment variables http_proxy and HTTPS_PROXY respectively. The config file takes precedence over environment variables.

file_urls

Since: 0.10.0

Optional boolean value (default false) indicating whether to allow fetching web pages from file URLs. When set to true, feed.config.url can be a URL using the file scheme to a local HTML file like: file:///home/wmoore/Documents/example.html. The path must be absolute.

feed.title

The title of the channel in the generated feed.

feed.filename

The output filename to write this feed to. Note: this is a filename only, not a path. It should not contain slashes. It will be written to the output directory.

feed.config.url

The URL of the web page to generate the feed from. The page at this address will be fetched processed to turn it into a feed.

feed.config.item

A CSS selector to select elements on the page that represent items in the feed. The other CSS selectors match elements inside the elements that this selector matches.

feed.config.heading

A CSS selector relative to item to an element that will supply the title for the item in the feed.

CSS selector relative to item to an element that will supply the link for the item in the feed.

Note: This element must have a href attribute.

Note: If not supplied rsspls will attempt to use the feed.config.heading selector as the link element for backwards compatibility with earlier versions. A warning message will be emitted in this case. It is recommended to specify the link selector explicitly.

feed.config.summary

Optional CSS selector relative to item that will supply the content of the RSS item. This value may be a single CSS selector, or an array of CSS selectors.

The CSS selectors may also include a comma separated list of elements to match. For example: summary = "p, blockquote" will match p or blockquote elements, adding them to the RSS feed in the order then are encountered in the HTML document.

The array form of summary allows the order of the matched elements to be controlled, enabling elements to be added to the feed in a different order to the source HTML document. For example, summary = ["p", "blockquote"] causes rsspls to make a pass over the source HTML document, adding p elements to the feed, followed by a pass adding blockquote elements to the feed.

feed.config.date

The optional date key in the configuration can be a string or a table. If it’s a string then it’s used as CSS selector relative to item to find the element containing the date and rsspls will attempt to automatically parse the value.

If automatic parsing fails you can manually specify the format using the table form of date, which looks like this:

[feed.config.date]
selector = "time" # required
type = "Date"
format = "[day padding:none]/[month padding:none]/[year]" # will parse 1/2/1934 style dates

If the element matched by the date selector is a <time> element then rsspls will first try to parse the value in the datetime attribute if present. If the attribute is missing or the element is not a time element then rsspls will use the supplied format or attempt automatic parsing of the text content of the element.

feed.config.date.selector

CSS selector relative to item that supples the publication date of the RSS item.

feed.config.date.type

Optional type of value being parsed. Either Date or DateTime.

type is Date when you want to parse just a date. Use DateTime if you’re parsing a date and time with the format. Defaults to DateTime.

feed.config.date.format

Format description using the syntax described on this page: https://time-rs.github.io/book/api/format-description.html of how to parse the date.

feed.config.media

Optional CSS selector relative to item that supplies media content (audio, video, image) to be added as an RSS enclosure.

Note: The media URL must be given by the src or href attribute of the selected element.

Note: Currently if the item does not match the media selector then it will be skipped.

Hosting, Updating, and Subscribing

In order to have the feeds update you will need to arrange for rsspls to be run periodically. You might do this with cron, systemd timers, or the Windows equivalent.

To subscribe to feeds you can run rsspls locally and use a feed reader that supports local file feeds. Or, more likely it is expected that rsspls will be run on a web server that is serving the directory the feeds are written to.

Logging

rsspls logs messages to stderr. Logging can be controlled by the RSSPLS_LOG environment variable. Log level and target module can controlled according to the env_logger documentation. For example, to enable debug logging for rsspls you would use:

RSSPLS_LOG=rsspls=debug

The supported log levels are:

The default log level is info.

Caveats & Error Handling

rsspls just fetches and parses the HTML of the web page you specify. It does not run JavaScript. If the website is entirely generated by JavaScript (such as Twitter) then rsspls will not work.

If errors are encountered processing the page due to invalid selectors, or missing elements an error message will be logged. If the error is non-recoverable rsspls will exit with a non-zero exit status.

If an error is encountered processing an item for the feed a warning will by logged and processing will continue with the next item. rsspls will still exit with success (0) in this case.

Caching

When websites respond with cache headers rsspls will make a conditional request on subsequent runs and will not regenerate the feed if the server responds with 304 Not Modified. Cache data is stored in $XDG_CACHE_HOME/rsspls, which defaults to ~/.cache/rsspls on UNIX-like systems or C:\Users\You\AppData\Local\rsspls on Windows.