"Discovering" of GDS-Instances

[sternenseemann]

For some Apps it's useful to know as many GDs Instances as possible, for example for a decentraliced search etc.
Therefore we may add something like a "Known GDS-Instances"-API which is managed and extended by each GDS. When it gets to know a new isntance it also extends its own list of GDS-Instances using the list of the new GDS-Instace.

[augustl]

Some kind of tracker system similar to BitTorrent would be great. Discoverability does require a central register machine, but like BitTorrent, the actual server being used can be pluggable, you can use multiple servers, etc.

I don't know a lot about BitTorrent, but perhaps the entire discovery part of BitTorrent can be completely reused?

[sternenseemann]

But Bittorrent is closed source, isn't it?

[augustl]

The bittorrent protocol is open: http://www.bittorrent.org/beps/bep_0003.html

I'm not sure about the license of the protocol, though.

[lukasbestle]

It's very important to decentralize this discovery process as well.
A possible solution would be that a GDS doesn't know about other GDSs until the user interacts with another one.
The admin of the local GDS could define if the URL of the GDS should be announced to other devices as well or if it should be kept private.
You could then ask any GDS about a list of public devices' URLs.
Just like DNS, these would be cached on every device, so the list gets bigger pretty fast.

[sternenseemann]

@vis7mac Yes, that was my idea, maybe I didn't point it out clearly.

[sternenseemann]

@augustl The Protocol document is public domain...

[DerZyklop]

@vis7mac @lukasepple great idea! But whats with obsolete URL's. Do obsolete URL's disappear automatically from the lists? Do GDS has to care about this issue, anyway? ...don't know how this works in case of DNS.

[lukasbestle]

@DerZyklop DNS has a TTL (time to live) time for each record, for example 3600 for "an hour" or 60 for "a minute".
After this time expires, every cache of this record must be deleted and updated (a DNS client would ask the server again, the server would ask another server, the other server would ask the domain's nameserver and so on).

This might be an issue for GDS, because it would take quite a lot network traffic and computing power, although we could use it in an amazing way: uptime monitoring.
There could be a "discovery and uptime" GDS app which could be installed voluntarily.
It would collect a list of domains of other GDS installations together with a mail address of the admin of the GDS hosted on another service and the information if the server is a "discovery and uptime" server.
The app would check every defined period of time if each of these domains is up and if a GDS lives there. If not, a mail is sent to the alternative email address of the user notifying him of the problem.
If it stays down for over a week or so, the domain is removed from the index until it gets into the index of at least one server again.

A possible problem: The alternative email address would get a lot of mails for each downtime if every server sent a mail. So all GDSs have to communicate to

• check if the domain is down from another point in the world as well
• send exactly one email from one server with an installed "discovery and uptime" server.

A requirement for this is that every single GDS with a "discovery and uptime" server knows of all the other ones.
This would be given after a specific amount of time (when each server communicated to at least one other server which has a connection to at least one other server etc.).

[lx4r]

@vis7mac Wow, that sounds like an awesome idea! "Discovery and Uptime" would be a really nice app if we manage to take care of the mail spam problem. But that should be possible ...
Maybe to even the amount of traffic (I know that sending mails doesn't consume much traffic but just to be fair) the first GDS that notices that another GDS is down could selecet a random GDS with the "Discovery and Uptime" app installed from the ones it knows and tell that one GDS to send the "Your server is down" mail.
Another usecase of the "Discovery and Uptime" app could be that if the GDSs knows their positions (e.g. Frankfurt in Germany) they could find out whether a server is only unreachable from one country or from everywhere.

[lukasbestle]

I guess the first server noticing this would select a random new one standing at a different position.
If for example 5 servers in a row notice a downtime, this one sends the mail.

But this does still not resolve the spam problem.

[lukasbestle]

Oh, hey: I just got an amazing idea how to solve this:
Let's say the server to monitor is named Rolf.
A few first "discovery and uptime" servers Rolf finds out about would be remembered by it. The database item of Rolf in the discovery cache of all "discovery and uptime" servers would now contain the following attributes:

• the URL of Rolf
• a mail address to notify the user when a downtime happens
• a list of 3 "mentoring" servers of Rolf (if the first one is down, the second one gets pinged as a backup etc.)

If another GDS with "discovery and uptime" now finds out that Rolf is down, it sends that information to another "discovery and uptime" server.
This would be repeated 5 times. If the server is up from at least one GDS, it all stops. If not, a "mentoring server" is pinged about that. It sends the mail to Rolf's owner about the issue and remembers that until Rolf is up again (after that, another email can be sent in the future).
This way, Rolf's owner can not get more than 3 mails (worst case if some mentoring servers are down from one part of the world).

This would generate a lot of traffic if there are many GDSs (the mentoring servers would get a lot of requests by kind of every "discovery and uptime" server).
So there would probably be dedicated fee-based monitoring services taking the role of a mentoring server.

I will write more about this idea in the concept markdown later on.

[lukasbestle]

So I actually wrote something structured about it. And it became a novel. :P
8683a3f

[bastianallgeier]

Please stop closing issues!! We need this discussions. This is still a concept and even if ideas sound great this is about writing them down first then thinking about implementations then building them and then reevaluate if they work together and if they help the general concept. This is not about playing: "ok I think I solved this" and closing stuff within a couple hours.

[sternenseemann]

Aye! You're right.

[lukasbestle]

Yep, that's totally right.
But the question is if we need to use issues for that. It's a good way to discuss stuff, but from time to time, someone has to collect that ideas and add them to the repository.

[bastianallgeier]

Absolutely!! Those discussions should always lead to additions or edits of the concept.

[lukasbestle]

I'm doing that in the moment for the current issues by linking to them in the Markdown.

[bastianallgeier]

That's perfect!!

[pxwrk]

Additional to @vis7mac "3 mentoring servers"-system – an idea to avoid multiple notifications to the Admin:

Isn't it possible that the 3 "mentoring" servers of Rolf make oneself known to each other? So if one of them gets a order to send a notification to the Rolf-Admin, it might also ask the others, if someone has already notified the Rolf-Admin.

And afterwards, the "mentoring" server might inform the others that a Notification was send.

[lukasbestle]

That's a pretty nice idea! I will add that to the Markdown ASAP.