Add support for DNS

[filips123]

This adds support for classic DNS. It closes #104 and it is part of #2049.

---

The plugin uses DNSPython library for getting DNS responses and DNSLink for resolving DNSLink records.

It uses a few DNS-over-HTTPS resolvers with OpenNIC TLD support, but it can also be configured to use other resolvers or system configuration. Results are also cached in files so domains will still work if the computer is offline.

There are already a few (my) domains that use DNS plugin which you can use for testing:

• py.zeroframe.oss
• js.zeroframe.oss

---

Note that because now plugin has additional dependencies, I had to change the configuration for Travis CI, GitLab CI, Docker and Vagrant to also install dependencies from (enabled) plugins. @HelloZeroNet ZeroBundle also needs to be updated to do this.

Also, DNSPython currently doesn't support DNS-over-HTTPS so I created PR rthalley/dnspython#393 with support for it. Until it is not merged and released, we have to use my temporary branch. Then you would need to update it to use a newly created version.

I also think that there this domain plugins aren't currently working if default Zeroname is disabled. I already reported this in #2128, when it was partly fixed (before that it was completely impossible to create domain plugin), but it appears that there are still problems if Zeroname is disabled. I wasn't able to test this with this plugin, but it needs to be investigated.

There are currently also some problems with CI. Tests works, but there is problem with linter (flake8). Because DNSPython needs to be loaded after Tor monkey-patching is completed, I needed to load them from a class method. I implemented that with importing modules inside a function and adding them to globals(). This works normally as modules are now availible at global context. However, flake8 doesn't like that and throws F821 undefined name errors. Is there any way to tell flake8 that dns and dnslink are actually existing names?

---

@HelloZeroNet @imachug @ValdikSS Can you test and rewiev this?

[purplesyringa]

Congrats, the tests are passing now.

[purplesyringa]

@filips123 Should this PR be merged if it doesn't work without Zeroname plugin?

[filips123]

@imachug It should be. Zeroname is still the default domain system and almost no one would disable it. But I think that this issue should be fixed quickly.

But I actually only tested this few months ago with ENS plugin (which uses same structure as this). I will also try to test this again now.

Update: I tested this again and I can confirm that it works only when Zeroname is enabled. When Zeroname is disabled, the site will try to load (it will open wrapper and page title will be set), but the actual page will load forever. See debug log here - the error is (almost) the same as in #2128. When Zeroname is enabled, it will work normally.

[filips123]

@HelloZeroNet Do you know what could be a reason for that error?

[HelloZeroNet]

I think this should be a separate plugin, because depending on the traditional dns system somehow defeat the decentralized and uncensorable goals of the project.

I will check why is it not working without zeroname

[filips123]

@HelloZeroNet I don't think so. I would agree with you if my plugins would be the only way for domains in ZeroNet. But since users can also use Zeroname (or any other domain plugin), it would actually make it more decentralized as users won't be forced to use one particular domain system.

[ghost]

Exactly! Multiple centralized solutions provides more decentralization than one centralized solution (and I would even say multiple solutions is better than one period). Of course, the best would be multiple decentralized solutions, but we have more work to get to that point.

[filips123]

Of course, the best would be multiple decentralized solutions, but we have more work to get to that point.

And it is actually not so easy to implement a really decentralized solution. Almost every third-party domain system for integration with ZeroNet (or any other project) would require either:

• Complete protocol implementation with locally stored data (which in case of blockchain-based systems means to have the whole blockchain stored locally).
• Data stored in a custom location (like Zeroname site).
• Data from a third-party provider (like Infura for Ethereum).

[filips123]

@HelloZeroNet @shortcutme Did you find why this plugin doesn't work without Zeroname?

[HelloZeroNet]

Yes, I pushed an update (Rev4238) that should make it work without the Zeroname plugin.

It happened because of the missing actionSiteMedia patching without the Zeroname plugin.
I have moved the domain resolving to the parsePath function and also added a short cache (10sec) for isDomain / resolveDomain function, because the parsePath function could be called multiple times for each http request.
43a5742
924a613

[filips123]

@HelloZeroNet Should I change anything in this plugin or it will simply work?

From your commits, I saw that you removed actionSiteMedia. Does this mean that the whole UiRequestPlugin is not needed anymore?

[ghost]

Hi @filips123 @HelloZeroNet
Look #2208 I don't see how _dnslink is better than a reverse proxy to /raw/. Problematic is the forced headers with ZeroNet. At least when proxy and _dnslink detected to "raw" ZeroNet should remove the headers frame etc... Unnecessary. Anyway content.json is exposed to the world which may be a problem if ZeroNet signs the same file two times! If that's the case the private key can be recovered. Right now I stick to reverse proxy. As mentioned on the link above..

[ghost]

@filips123 With reverse proxy I can use any TLD I want to host on ZeroNet. I don't really think this _dnslink will allow me to use for example:

zeronet.example.org. IN CNAME zeronet.gateway.example.org
_dnslink.zeronet.example.org. IN TXT dnslink=/raw/bitcoinaddress

That would be the goal no? The headers and sandbox in any case where /raw/ requested should be removed!

[d14na]

@filips123 i've been thinking about how to bridge ENS to Zeronet for a while now. With the new ENS system and 3,4,5-letter domains being auctioned now, I think the system will gain traction within the crypto community. What's the status of #2164? Is this working?

If so, then imo the next step is to bridge back to the clearnet for users NOT running the client. Cloudflare offers this for IPFS websites. Currently in the process of moving almost all of our front-end code over to IPFS. Honestly, it has its issues (mainly caching seems sketchy, but perhaps more seeding is needed), but overall its been working flawlessly for months.

You can read about it here, if you're unfamiliar:
https://www.cloudflare.com/distributed-web-gateway/

The idea is, our proxy at 0net.io would serve (as it does now) the files to Clearnet users, however, they would be able to use https://example.eth instead of https://0net.io/example.bit. Obviously, this would NOT be limited to just .eth, I'm thinking .xyz too (they have a very progressive admin team).

While still making improvements to the proxy's underlying infrastructure, its realatively stable at this point https://status.0net.io. Let me know if you'd be interested in adding that support to your plugin. And, well, how you think we'd go about implementing that into the current proxy's code.

Regardless, good luck with this effort. Don't hesitate to reach for help testing, debugging, whatever. cheers!

[ghost]

@d14na This has nothing to do with dnslink on Zeronet far as I read your comment. Lastly forget 0net.io because that is a big trash. If you want to proxy clearnet to ZeroNet you will need your own servers for front and back-end!!!!!

[purplesyringa]

A private key can only be recovered if you sign something twice using the same random number, which is next to impossible

[filips123]

@HelloZeroNet I bundled all plugin's dependencies (DNSPython and DNSLink) into ZIP file along with plugin's source code. That ZIP file is then added to Python's path (Python can load modules from ZIP) in case if dependencies are not already installed.

I also included support for DNS-reverse-proxy (implemented by @imachug) which enables you to create a site which is hosted on ZeroNet and access it like normal website. To do this, you need to create DNSLink TXT record that points to site address and CNAME/A/AAAA that points to ZeroNet instance. In case of local ZeroNet instance, the site can be accessed at example.com:43110. In case of public ZeroNet proxy, site can be accessed directly at example.com (because most proxies host ZeroNet on port 80). Note that some sites (like ZeroHello, probably also other official sites with dynamic links to other sites) are not yet ready for this as they make incorrect links to other sites. This needs to be fixed in their end.

[filips123]

@HelloZeroNet @shortcutme Can you check and merge this?

Dependencies are now bundles with source code (and all licenses are compatible) so there shouldn't be any problem with using this plugin.

This would allow accessing ZeroNet sites from classic DNS domains which is useful in case if site author already has domain (or want to use free OpenNIC domain) and don't want to register another Namecoin domain.

And it would also allow accessing ZeroNet as reverse-proxy. So if the domain has CNAME/A/AAAA record that points to ZeroNet instance and TXT record which contains the address of the site, the site will be directly accessible from that domain.

In case of local instances, this will have to include port 43110. Solution for this would be to use 127.43.11.0:80 as default host of ZeroNet UI. This will also enable viewing ZeroNet sites with local instances from on port 80 while still keeping ZeroNet separated from other services which may run on port 80 (as it would use 127.43.11.0 which is another loopback IP). IPs in 127.*.*.* should resolve to loopback in most cases, but I think that in macOS, the only loopback address is 127.0.0.1. This means that for it, we would need to manually assign new IP to loopback interface, which probably requires root permissions.

But this is actually not so important for this PR so it would be great if you can merge it.

And even if classic DNS is centralized, adding it as additional, but enabled by default, domain system won't defeat decentralized goals of ZeroNet, as users will actually have more choice between domain systems. And with ENS support, it would make it even more decentralized.

[HelloZeroNet]

Ok, I'm not against it, I will do some tests later on it.

[filips123]

My PR for DNS-over-HTTPS support has been merged, so we don't need to depend on my fork anymore. However, we still need to use Git-based version, as changes still aren't released in new version.

[filips123]

@imachug I tested DNS reverse proxy with my public ZeroNet proxy and problem is that sometimes cookies are not sent to WebSocket connection which means that connection fails with Multiuser plugin. Do you think it would be possible to send that cookie to WebSocket connection even if it is in a different origin than the website?

Also, zero button links to zero/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D but this can probably be fixed easily with some condition.. There might also be some other problems with other features that try to connect to wrong URL.

Also, can you check if there can be any possible security problems with accessing WebSocket from a different origin and such?

[purplesyringa]

Do you think it would be possible to send that cookie to WebSocket connection even if it is in a different origin than the website?

AFAIK there's no way to add a cookie to another site. The only solution I've managed to come up with is sending the cookie via the websocket, say, in the first message. @HelloZeroNet Do you think that might work?

[purplesyringa]

Also, zero button links to zero/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D but this can probably be fixed easily with some condition..

Where do you want it to redirect to? Probably the root of the site?

[purplesyringa]

Also, can you check if there can be any possible security problems with accessing WebSocket from a different origin and such?

I think that's safe. We're using wrapper_key as a security measure so there's no problem as long as the key verification is in action.

[filips123]

AFAIK there's no way to add a cookie to another site.

In non-web-based WebSocket clients (in Node.js and Python) you can set cookies. However, I'm not sure if browsers also allow this.

So yes, if sending cookies is not possible, it should be sent either in URL or in first message.

Where do you want it to redirect to? Probably the root of the site?

Yes.

[purplesyringa]

In non-web-based WebSocket clients (in Node.js and Python) you can set cookies. However, I'm not sure if browsers also allow this.

You can set cookies in browsers too but only if they are same-origin.

[filips123]

There is one other thing. Is there any reason why WebSocket requests are made to different origin that is specified in server_url instead of accessing from same origin? I think it may be with some issues with old Chrome proxy extension with WebSockets but I'm not sure?

[purplesyringa]

I think there was a reason but I can't remember now. @HelloZeroNet?

[HelloZeroNet]

Yeah PAC-based proxy configs/extensions does not work with websocket. So when you access http://zeronet.bit you still have to connect http://127.0.0.1:43110/Websocket

[purplesyringa]

What is the state of this PR?

[filips123]

Just accessing DNS domains works.

However, DNS reverse proxy with Multiuser plugin sometimes doesn't because of cookies and CORS.

[filips123]

I have idea how this cross-origin cookie/multiuser handling could be implemented:

1. Wrapper/client normally sets cookie and tries to use it on WebSocket connection, like it does now.

2. When WebSocket server (Multiuser plugin) receives request, it checks for cookie. If it exists, it uses it to get account/address, also like it does now.

3. However, if cookie isn't set (probably because of CORS), instead of failing with error, it immediately sends some message/request/action/command to wrapper/client for user address.

4. Wrapper/client listens for that message, and if it receives it, it sends user address as response. Then it can start sending commands normally.

5. Once server receives response, it uses it as account. If it receives any other commands before response (and response was requested), it fails with error that user isn't authenticated (similarly to what currently happens).

Advantage of this is that if cookies are sent in first place, there isn't any need for any additional message, which means that it shouldn't affect speed. Additionally, it stays compatible with all current clients.

[purplesyringa]

Wrapper/client listens for that message, and if it receives it, it sends user address as response. Then it can start sending commands normally.

A small fix: user masterseed, not address.

[filips123]

Master seed or master address? Cookie currently contains master address, so that response should probably also use it. And given that it will probably be used in somewhere in browser's storage (probably cookie just like now, but client just reads it from cookie and sends it to server if needed), it probably won't be good to store sensitive information such as master seed.

[purplesyringa]

Daaaaaamn, and that's what we're calling "secure"... /cc @HelloZeroNet