nostr-nomad

nostr-nomad migrates content from Substack to Nostr relays.
This project is a collaboration between Natalie and Alex.

Supports:

• Short-form content (event kind:1; NIP-1) — such as "Tweet"-style messages (no formatting).
• Long-form content (event kind:30023; NIP-23) — such as blog posts (with Markdown formatting).

Features:

• Maintains a local archive of all exported post text, associated images, and publishing metadata.
• Optionally re-hosts images to move media off Substack’s CDN
  • Supported hosts: Imgur (anonymous uploads) and any self-hosted HTTP file server running WALA

---

Requirements

All Python dependencies will be automatically installed into a virtual environment located at env/.

System Dependencies

You'll need Python 3.11.x to run the project.

• Check your Python version: python3.11 --version
• If the command is not found, please follow the instructions below to install it and check again:
  • Ubuntu/Debian:

    sudo apt update
    sudo apt install software-properties-common -y
    sudo add-apt-repository ppa:deadsnakes/ppa -y
    sudo apt update
    sudo apt install python3.11 python3.11-venv python3.11-dev -y

  • macOS:

    brew install python@3.11
    brew link --force python@3.11 # Makes 'python3.11' command accessible system-wide

To generate documentation, please install Texinfo and TeX first:

• Ubuntu/Debian: sudo apt install texinfo texlive -y
• macOS: brew install texinfo texlive

Make sure groff is installed to display the CLI manual: groff --version. If not installed, install it via:

• Ubuntu/Debian: sudo apt install groff
• macOS: brew install groff

Ready-to-use Codespace

This repo’s codespace is pre-configured with all dependencies through the devcontainer. Feel free to quickly try out nostr-nomad there.

---

How to Use

1. Clone the Repository
   Clone the repository and navigate into the project directory:

   git clone https://github.com/alx-sch/nostr-nomad nostr-nomad && cd nostr-nomad
   

2. Provide Substack Export

• Export your Substack data (check here).
• Optional: After unzipping the archive, you can remove any posts you don’t want to migrate by deleting their corresponding .html files in the posts/ folder. Make sure to also remove these posts from the posts.csv file.
• Place the export zip file or directory into user_entries/export/. It should contain a posts.csv file and a posts/ directory with your .html files.

3. Set Configurations Provide the following information in the user_entries/config.ini file:

• Private key: Enter your Nostr private key in hex or nsec format, or set it to x to generate a random key at runtime. This key is used only to sign Nostr events locally and is never shared.
  ⚠️ Note: Some relays may require prior authorization and might reject events from unknown keys.
• Post type: Choose what kind of post you want to publish:
  • note — short, unformatted notes (like tweets)
  • blog — longer, formatted blog-style posts
• Relays: List the WebSocket URLs of the relays you want to publish to.
• Image hosting (optional): Choose how you'd like to handle images by setting image_host to one of the following:
  • substack – Keep the original image URLs from Substack
  • imgur – Upload images to imgur.com
  • wala – Upload images to your own WALA server (a self-hosted image store)
    If you choose wala, make sure to set your wala_url.
    If you choose imgur, you'll need to provide your imgur_client_id.

4. Run nostr-nomad

   • Build and Run
     make — Sets up the virtual environment, installs dependencies, and runs the tool.
   • Generate Documentation (WIP)
     make docs — Builds documentation in various formats: README.md, HTML, PDF.
     make clean-docs — Removes generated documentation files.
   • View Man Page (WIP)
     make man — Displays a terminal-friendly manual for quick command reference.
   • Clean Build Artifacts
     make clean — Removes the virtual environment and temporary files.
   • Full Cleanup
     make clean-all — Removes everything: environment, docs, cache/archive, and Substack export data.

5. Review Cache & Archive
   After running, nostr-nomad maintains a local cache in cache/:

   • posts.json — Tracks which posts have been published to which relays (prevents duplicates on re-runs).
   • images.json — Maps original image URLs to re-hosted URLs and local file paths.
   • imgur_delete_hash.json — Stores Imgur delete tokens so you can remove uploaded images later.
   • posts/ and images/ — Local copies of downloaded post content and images.

---

⚠️ Troubleshooting

If you run into issues like:

• ModuleNotFoundError
• Environment seems broken or misconfigured
• Tool fails to run right after a make

Try resetting the environment:

make uninstall
make install

• Relay publishing failures (timeout, rejected events)

  • Verify relay URLs are correct and reachable.
  • Some relays reject rapid-fire publishing; increase the delay between posts.
  • Some relays require your public key to be pre-authorized.

• Imgur upload errors

  • Double-check your imgur_client_id in config.ini.
  • Imgur rate-limits anonymous uploads; wait and retry if you hit a 429 response.

---

Known Limitations

• Short-form vs. long-form: Short-form notes (kind:1) strip all formatting. Use blog mode if you need Markdown headings, links, or embedded images.
• Client compatibility: Not all Nostr clients support long-form content (kind:30023) or preview all image formats (e.g., most support .jpeg but not all support .avif).
• Relay speed: Some relays reject events published in quick succession — increase the delay between posts if you experience dropped messages.

Example Output

The tool reports per-relay status for each post:

• OK — Successfully published.
• Error — Relay unreachable (check the URL or ensure the relay is running).
• Already published. Skipping this relay. — Post was previously sent to this relay (tracked via the local cache).

---

Setting up a Local Nostr Relay for Testing

Before publishing to public Nostr relays, consider testing your setup with a local relay. This lets you explore how nostr-nomad works without impacting the broader network. Below is a setup guide for nostr-rs-relay, but any other compatible relay should work, too.

1. Install Rust
   The first step is to install Rust. This is needed to compile and run the nostr-rs-relay project.

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

   To make rustc and its package manager cargo available in your terminal, run:

   source $HOME/.cargo/env

   You can verify that the installation was successful by checking the versions:

   rustc --version
   cargo --version

2. Install the Protocol Buffers Compiler
   The build process for nostr-rs-relay requires the Protocol Buffers compiler (protoc). You can install it on Ubuntu with:

   sudo apt install protobuf-compiler

   After installation, confirm that it's available by checking the version:

   protoc --version

3. Clone the nostr-rs-relay Repository
   Now, you’ll clone the nostr-rs-relay repository to your local machine. This repository contains the source code for the relay server.

   git clone https://github.com/scsibug/nostr-rs-relay.git nostr-rs-relay && cd nostr-rs-relay

4. Build the Relay Server
   Compile the project with the cargo build tool; this might take some time.

   cargo build --release

5. Edit Configuration for the Local Relay
   In your config.toml file, change the following fields to set up a local Nostr relay:

   [network]
   address = "127.0.0.1"  # Or any other appropriate loopback IP, 127.0.0.1 is usually used as localhost.
   port = 8081            # Use any available port; 8081 is commonly used for development/testing.

   Since this is for local testing, we will not enable WSS (secure WebSocket). This avoids the need to generate SSL/TLS certificates or configure encryption — keeping the setup simple and focused. The relay will run at: ws://127.0.0.1:8081 or simply ws://localhost:8081 (confirm that localhost points to 127.0.0.1 by running ping localhost).

6. Starting the Nostr Relay
   Before running the relay, you need to create the database directory first (mkdir db) in the root of the nostr-rs-relay repository. The relay will store data such as events and connections here.
   Once the database directory is created, you can start the relay by running the following command. This will provide detailed log output:

   RUST_LOG=debug ./target/release/nostr-rs-relay -c config.toml -d db

   The relay should begin running, and you’ll see status updates in the terminal. This confirms that the relay is active and listening for WebSocket connections. Leave the terminal window open to keep the relay running. To interact with the relay, use nostr-nomad or other testing methods through a separate terminal window.

   

7. Checking Messages of the Local Nostr Relay
   You can use any Nostr client that allows for the inclusion of local relays. For example:

   • Gossip
   • iris

   For adding the local Nostr relay to Gossip, refer to this helpful blog post.

   For a quick setup of the iris client, you can use the standalone desktop release. After installation and signing up:

   • Go to Settings -> Network.
   • Add ws://localhost:8081 as a new relay.
   • Uncheck any other relays you might have subscribed to (to keep the feed focused on the local relay).
   • Check the feed (house symbol).

   If everything is set up correctly, you should now see events published to your local Nostr relay.

---

Acknowledgements

We'd like to thank lash for providing the excellent blog post, which helped us set up local relays and figure out how to send events to them. Lash has also been a mentor throughout this project.