Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zeno doesn't start with get list and an empty line in list #146

Open
CorentinB opened this issue Sep 9, 2024 · 2 comments
Open

Zeno doesn't start with get list and an empty line in list #146

CorentinB opened this issue Sep 9, 2024 · 2 comments
Labels
bug Something isn't working good first issue Good for newcomers P2 high priority

Comments

@CorentinB
Copy link
Collaborator

If you use get list with a seeds list that contain an empty line, Zeno won't start crawling.

@CorentinB CorentinB added bug Something isn't working good first issue Good for newcomers P2 high priority labels Sep 9, 2024
@yzqzss
Copy link
Contributor

yzqzss commented Sep 13, 2024

URL, err := url.Parse(scanner.Text())

https://pkg.go.dev/net/url#Parse

Trying to parse a hostname and path without a scheme is invalid but may not necessarily return an error, due to parsing ambiguities.

https://go.dev/play/p/gY82hrbRTh2


We probably need a better url parser.

@CorentinB
Copy link
Collaborator Author

URL, err := url.Parse(scanner.Text())

https://pkg.go.dev/net/url#Parse

Trying to parse a hostname and path without a scheme is invalid but may not necessarily return an error, due to parsing ambiguities.

https://go.dev/play/p/gY82hrbRTh2

We probably need a better url parser.

Agreed. On the same topic.. I found Zeno to be absolutely unusable (I guess it's the queue) when queuing a 8.5M long seeds list. You might want to try it out if you're interested..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers P2 high priority
Projects
None yet
Development

No branches or pull requests

2 participants