xurlfind3r is a command-line interface (CLI) utility to find domain's known URLs passively from AlienVault's Open Threat Exchange, Common Crawl, Github, Intelligence X, URLScan, and the Wayback Machine.

• Features
• Installation
  • Install release binaries
  • Install source
    • go install ...
    • go build ... the development Version
• Post Installation
• Usage
• Contribution
• Licensing

• Fetches known URLs from AlienVault's OTX, Common Crawl, URLScan, Github, Intelligence X and the Wayback Machine.
• Parse URLs from robots.txt snapshots on the Wayback Machine.
• Parse URLs from webpages snapshots on the Wayback Machine.

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

• Clone the repository

   git clone https://github.com/hueristiq/xurlfind3r.git 

• Build the utility

   cd xurlfind3r/cmd/xurlfind3r && \
   go build .

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xurlfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

xurlfind3r will work right after installation. However, Github and Intelligence X require API keys to work. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format.

Example:

version: 0.1.0
sources:
    - commoncrawl
    - github
    - intelx
    - otx
    - urlscan
    - wayback
keys:
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000

To display help message for xurlfind3r use the -h flag:

xurlfind3r -h

help message:

                 _  __ _           _ _____      
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __ 
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
 >  <| |_| | |  | |  _| | | | | (_| |___) | |
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| v0.1.0

A CLI utility to find domain's known URLs.

USAGE:
  xurlfind3r [OPTIONS]

TARGET:
  -d, --domain string             target domain
      --include-subdomains bool   include domain's subdomains

SOURCES:
      --list-sources bool         list available sources
 -s   --sources strings           comma(,) separated sources to use (default: commoncrawl,github,intelx,otx,urlscan,wayback)

CONFIGURATION:
      --skip-wayback-robots bool  skip parsing wayback robots.txt snapshots
      --skip-wayback-source bool  skip parsing wayback source code snapshots

OUTPUT:
  -m, --monochrome                no colored output mode
  -o, --output string             output file to write found URLs
  -v, --verbosity                 debug, info, warning, error, fatal or silent (default: info)

Issues and Pull Requests are welcome! Check out the contribution guidelines.

This utility is distributed under the MIT license.