2 changes: 1 addition & 1 deletion 2 .github/ISSUE_TEMPLATE/bug-report.md

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,7 8,7 @@ assignees: ''
		---
		**Describe the bug**
		A clear and concise description of what the bug is. Include the version of urlfind3r you are using.
		A clear and concise description of what the bug is. Include the version of hqurlfind3r you are using.
		**Steps To Reproduce**
		Steps to reproduce the behavior: e.g. The complete command you used/ran.
Expand Down

2 changes: 1 addition & 1 deletion 2 .github/workflows/build.yml

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -24,4 24,4 @@ jobs:
		-
		name: Build
		run: go build -v .
		working-directory: ./cmd/urlfind3r
		working-directory: ./cmd/hqurlfind3r

2 changes: 1 addition & 1 deletion 2 .gitignore

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 1,6 @@
		# Executable
		cmd/urlfind3r/urlfind3r
		cmd/hqurlfind3r/hqurlfind3r
		# Notes
Expand Down

4 changes: 2 additions & 2 deletions 4 .goreleaser.yaml

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -4,8 4,8 @@ before:
		builds:
		-
		binary: urlfind3r
		main: cmd/urlfind3r/main.go
		binary: hqurlfind3r
		main: cmd/hqurlfind3r/main.go
		goos:
		- linux
		- windows
Expand Down

2 changes: 1 addition & 1 deletion 2 .vscode/settings.json

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,5 1,5 @@
		{
		"yaml.schemas": {
		"https://json.schemastore.org/github-workflow.json": "file:///home/enenumxela/Projects/hueristiq/open-source-projects/urlfind3r/.github/workflows/build.yml"
		"https://json.schemastore.org/github-workflow.json": "file:///home/enenumxela/Projects/hueristiq/open-source-projects/hqurlfind3r/.github/workflows/build.yml"
		}
		}

44 changes: 22 additions & 22 deletions 44 README.md

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 1,6 @@
		# urlfind3r
		# hqurlfind3r
		[](https://github.com/hueristiq/urlfind3r/releases)  [](https://github.com/hueristiq/urlfind3r/issues?q=is:issue is:open) [](https://github.com/hueristiq/urlfind3r/issues?q=is:issue is:closed) [](https://github.com/hueristiq/urlfind3r/blob/master/LICENSE) [](https://twitter.com/itshueristiq)
		[](https://github.com/hueristiq/hqurlfind3r/releases)  [](https://github.com/hueristiq/hqurlfind3r/issues?q=is:issue is:open) [](https://github.com/hueristiq/hqurlfind3r/issues?q=is:issue is:closed) [](https://github.com/hueristiq/hqurlfind3r/blob/master/LICENSE) [](https://twitter.com/itshueristiq)
		A passive reconnaissance tool for known URLs discovery - it gathers a list of URLs passively using various online sources.
Expand Down Expand Up		@@ -33,28 33,28 @@ A passive reconnaissance tool for known URLs discovery - it gathers a list of UR
		### From Binary
		You can download the pre-built binary for your platform from this repository's [releases](https://github.com/hueristiq/urlfind3r/releases/) page, extract, then move it to your `$PATH`and you're ready to go.
		You can download the pre-built binary for your platform from this repository's [releases](https://github.com/hueristiq/hqurlfind3r/releases/) page, extract, then move it to your `$PATH`and you're ready to go.
		### From Source
		urlfind3r requires **go1.17 ** to install successfully. Run the following command to get the repo
		hqurlfind3r requires **go1.17 ** to install successfully. Run the following command to get the repo
		```bash
		go install -v github.com/hueristiq/urlfind3r/cmd/urlfind3r@latest
		go install -v github.com/hueristiq/hqurlfind3r/cmd/hqurlfind3r@latest
		```
		### From Github
		```bash
		git clone https://github.com/hueristiq/urlfind3r.git && \
		cd urlfind3r/cmd/urlfind3r/ && \
		go build; mv urlfind3r /usr/local/bin/ && \
		urlfind3r -h
		git clone https://github.com/hueristiq/hqurlfind3r.git && \
		cd hqurlfind3r/cmd/hqurlfind3r/ && \
		go build; mv hqurlfind3r /usr/local/bin/ && \
		hqurlfind3r -h
		```
		## Post Installation
		urlfind3r will work after [installation](#installation). However, to configure urlfind3r to work with certain services - currently github - you will need to have setup API keys. The API keys are stored in the `$HOME/.config/urlfind3r/conf.yaml` file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these services.
		hqurlfind3r will work after [installation](#installation). However, to configure hqurlfind3r to work with certain services - currently github - you will need to have setup API keys. The API keys are stored in the `$HOME/.config/hqurlfind3r/conf.yaml` file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these services.
		Example:
Expand All		@@ -81,20 81,20 @@ keys:
		**DiSCLAIMER:** fetching urls from github is a bit slow.
		```bash
		urlfind3r -h
		hqurlfind3r -h
		```
		This will display help for the tool.
		```
		_ __ _ _ _____
		_ _ _ __| |/ _(_)_ __ __| |___ / _ __
		| | | | '__| | |_| | '_ \ / _` | |_ \| '__|
		| |_| | | | | _| | | | | (_| |___) | |
		\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| 1.8.0
		_ _ __ _ _ _____
		| |__ __ _ _ _ _ __| |/ _(_)_ __ __| |___ / _ __
		| '_ \ / _` | | | | '__| | |_| | '_ \ / _` | |_ \| '__|
		| | | | (_| | |_| | | | | _| | | | | (_| |___) | |
		|_| |_|\__, |\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| v1.8.0
		|_|
		USAGE:
		urlfind3r [OPTIONS]
		hqurlfind3r [OPTIONS]
		OPTIONS:
		-d, --domain domain to fetch urls for
Expand All		@@ -113,21 113,21 @@ OPTIONS:
		#### Basic
		```bash
		urlfind3r -d tesla.com
		hqurlfind3r -d tesla.com
		```
		#### Regex filter URLs
		```bash
		urlfind3r -d tesla.com -f ".(jpg|jpeg|gif|png|ico|css|eot|tif|tiff|ttf|woff|woff2)"
		hqurlfind3r -d tesla.com -f ".(jpg|jpeg|gif|png|ico|css|eot|tif|tiff|ttf|woff|woff2)"
		```
		#### Include Subdomains' URLs
		```bash
		urlfind3r -d tesla.com -iS
		hqurlfind3r -d tesla.com -iS
		```
		## Contribution
		[Issues](https://github.com/hueristiq/urlfind3r/issues) and [Pull Requests](https://github.com/hueristiq/urlfind3r/pulls) are welcome!
		[Issues](https://github.com/hueristiq/hqurlfind3r/issues) and [Pull Requests](https://github.com/hueristiq/hqurlfind3r/pulls) are welcome!

8 changes: 4 additions & 4 deletions 8 cmd/urlfind3r/main.go → cmd/hqurlfind3r/main.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -12,8 12,8 @@ import (
		"strconv"
		"strings"
		"github.com/hueristiq/urlfind3r/internal/configuration"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r"
		"github.com/hueristiq/hqurlfind3r/internal/configuration"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r"
		"github.com/logrusorgru/aurora/v3"
		)
Expand Down Expand Up		@@ -53,7 53,7 @@ func init() {
		banner()
		h := "USAGE:\n"
		h = " urlfind3r [OPTIONS]\n"
		h = " hqurlfind3r [OPTIONS]\n"
		h = "\nOPTIONS:\n"
		h = " -d, --domain domain to fetch urls for\n"
Expand Down Expand Up		@@ -119,7 119,7 @@ func main() {
		fmt.Println("")
		}
		runner := urlfind3r.New(&urlfind3r.Options{
		runner := hqurlfind3r.New(&hqurlfind3r.Options{
		FilterRegex: options.FilterRegex,
		SourcesToUse: options.SourcesToUse,
		SourcesToExclude: options.SourcesToExclude,
Expand Down

2 changes: 1 addition & 1 deletion 2 go.mod

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,4 1,4 @@
		module github.com/hueristiq/urlfind3r
		module github.com/hueristiq/hqurlfind3r
		go 1.18
Expand Down

4 changes: 2 additions & 2 deletions 4 go.sum

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -14,8 14,8 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
		github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
		github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
		github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
		github.com/hueristiq/urlfind3r v1.8.0 h1:xIjDNqRljRx7VAkOYl g0khoA 0m5hGnC8cMdwbuIYc=
		github.com/hueristiq/urlfind3r v1.8.0/go.mod h1:cxkL0lxiiMz5nLEEzjPoSS0dz8JEfqIqjXPzb1BNx2U=
		github.com/hueristiq/hqurlfind3r v1.8.0 h1:xIjDNqRljRx7VAkOYl g0khoA 0m5hGnC8cMdwbuIYc=
		github.com/hueristiq/hqurlfind3r v1.8.0/go.mod h1:cxkL0lxiiMz5nLEEzjPoSS0dz8JEfqIqjXPzb1BNx2U=
		github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF rwdDfMAkV7OtwuqBVzrE8GR6GFx wExME=
		github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
		github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn 9EI=
Expand Down

18 changes: 9 additions & 9 deletions 18 internal/configuration/configuration.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,8 7,8 @@ import (
		"path"
		"strings"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		"gopkg.in/yaml.v3"
		)
Expand Down Expand Up		@@ -46,12 46,12 @@ const (
		var (
		BANNER string = fmt.Sprintf(`
		_ __ _ _ _____
		_ _ _ __| |/ _(_)_ __ __| |___ / _ __
		| | | | '__| | |_| | '_ \ / _` "`" ` | |_ \| '__|
		| |_| | | | | _| | | | | (_| |___) | |
		\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| %s
		`, VERSION)
		_ _ __ _ _ _____
		| |__ __ _ _ _ _ __| |/ _(_)_ __ __| |___ / _ __
		| '_ \ / _` "`" ` | | | | '__| | |_| | '_ \ / _` "`" ` | |_ \| '__|
		| | | | (_| | |_| | | | | _| | | | | (_| |___) | |
		|_| |_|\__, |\__,_|_| |_|_| |_|_| |_|\__,_|____/|_| v%s
		|_|`, VERSION)
		)
		// ParseCLIOptions parse the command line flags and read config file
Expand All		@@ -61,7 61,7 @@ func ParseCLIOptions(options *CLIOptions) (parsedOptions *Options, err error) {
		return
		}
		configPath := directory "/.config/urlfind3r/conf.yaml"
		configPath := directory "/.config/hqurlfind3r/conf.yaml"
		parsedOptions = &Options{
		Domain: options.Domain,
Expand Down

18 changes: 9 additions & 9 deletions 18 pkg/urlfind3r/passive/agent.go → pkg/hqurlfind3r/passive/agent.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -6,15 6,15 @@ import (
		"regexp"
		"sync"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/commoncrawl"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/github"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/intelx"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/otx"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/urlscan"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/wayback"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping/sources/waybackrobots"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/commoncrawl"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/github"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/intelx"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/otx"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/urlscan"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/wayback"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping/sources/waybackrobots"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		// Agent is a struct for running passive url collection for a given host.
Expand Down

2 changes: 1 addition & 1 deletion 2 pkg/urlfind3r/scraping/sources.go → pkg/hqurlfind3r/scraping/sources.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,6 1,6 @@
		package scraping
		import "github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		import "github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		// URL is a result structure returned by a source
		type URL struct {
Expand Down

4 changes: 2 additions & 2 deletions 4 ...raping/sources/commoncrawl/commoncrawl.go → ...raping/sources/commoncrawl/commoncrawl.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,8 7,8 @@ import (
		"io/ioutil"
		"sync"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		type Source struct{}
Expand Down

4 changes: 2 additions & 2 deletions 4 ...lfind3r/scraping/sources/github/github.go → ...lfind3r/scraping/sources/github/github.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -12,8 12,8 @@ import (
		"strings"
		"time"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		"github.com/tomnomnom/linkheader"
		)
Expand Down

0 ...r/scraping/sources/github/tokenmanager.go → ...r/scraping/sources/github/tokenmanager.go

Show comments

View file Edit file Delete file

File renamed without changes.

4 changes: 2 additions & 2 deletions 4 ...lfind3r/scraping/sources/intelx/intelx.go → ...lfind3r/scraping/sources/intelx/intelx.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,8 8,8 @@ import (
		"log"
		"time"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		jsoniter "github.com/json-iterator/go"
		)
Expand Down

4 changes: 2 additions & 2 deletions 4 pkg/urlfind3r/scraping/sources/otx/otx.go → pkg/hqurlfind3r/scraping/sources/otx/otx.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -5,8 5,8 @@ import (
		"fmt"
		"io/ioutil"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		type Source struct{}
Expand Down

4 changes: 2 additions & 2 deletions 4 ...ind3r/scraping/sources/urlscan/urlscan.go → ...ind3r/scraping/sources/urlscan/urlscan.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -5,8 5,8 @@ import (
		"fmt"
		"io/ioutil"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		type response struct {
Expand Down

4 changes: 2 additions & 2 deletions 4 ...ind3r/scraping/sources/wayback/wayback.go → ...ind3r/scraping/sources/wayback/wayback.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -6,8 6,8 @@ import (
		"net/url"
		"strings"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		type Source struct{}
Expand Down

4 changes: 2 additions & 2 deletions 4 ...ng/sources/waybackrobots/waybackrobots.go → ...ng/sources/waybackrobots/waybackrobots.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -11,8 11,8 @@ import (
		"sync"
		"github.com/enenumxela/urlx/pkg/urlx"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		type Source struct{}
Expand Down

2 changes: 1 addition & 1 deletion 2 pkg/urlfind3r/scraping/utils.go → pkg/hqurlfind3r/scraping/utils.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
Expand Up		@@ -4,7 4,7 @@ import (
		"strings"
		"github.com/enenumxela/urlx/pkg/urlx"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		func NormalizeURL(URL string, scope session.Scope) (string, bool) {
Expand Down

0 pkg/urlfind3r/session/session.go → pkg/hqurlfind3r/session/session.go

Show comments

View file Edit file Delete file

File renamed without changes.

10 changes: 5 additions & 5 deletions 10 pkg/urlfind3r/sigurlfind3r.go → pkg/hqurlfind3r/sigurlfind3r.go

Show comments

View file Edit file Delete file

Original file line number	Diff line number	Diff line change
		@@ -1,4 1,4 @@
		package urlfind3r
		package hqurlfind3r
		import (
		"context"
Expand All		@@ -7,9 7,9 @@ import (
		"regexp"
		"github.com/enenumxela/urlx/pkg/urlx"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/passive"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/scraping"
		"github.com/hueristiq/urlfind3r/pkg/urlfind3r/session"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/passive"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/scraping"
		"github.com/hueristiq/hqurlfind3r/pkg/hqurlfind3r/session"
		)
		// Runner is an instance of url collection client used
Expand Down Expand Up		@@ -91,7 91,7 @@ func (runner *Runner) Run(ctx context.Context, domain string) (URLs chan scrapin
		if _, exists := deDupMap[key]; exists {
		for parameter := range parsedURL.Query() {
		if _, exists := deDupMap[key][parameter]; !exists {
		deDupMap[key][parameter] = []string{"urlfind3r"}
		deDupMap[key][parameter] = []string{"hqurlfind3r"}
		unique = true
		}
		}
Expand Down