Pool/Miner stats stop working

[mooleshacat]

@leshacat
Member
leshacat commented on 1 Nov 2018 •
When an invalid miner address makes it into the redis database, the miner stats and pool stats will stop working.

This is because when the browser is making the request to the API for live data, the API stumbles across an invalid address, and attempts to retrieve data for it - causing an exception, and halting the API response dead in it's tracks.

Short term fix is to just clear redis data with redis-cli FLUSHALL; unfortunately this clears all payment data.

Long term, this issue needs to be addressed. https://github.com/foxer666/node-open-mining-portal/issues/143 may fix it.

Solution is to validate every worker address that the API requests information for, and only request and return information for valid workers.

Also API data retrieval is very slow. This can be due to https://github.com/foxer666/node-open-mining-portal/issues/143 as well.

Lastly, I would ensure that JsonGet has some timeout set (even at 120 sec)

Duplicate issues:
https://github.com/foxer666/node-open-mining-portal/issues/150
zone117x/node-open-mining-portal#644
SkinnyPeteTheGiraffe#10

Samples:
https://miningpool.easyx.info/stats/mogwaicoin (charts not loading)
https://miningpool.easyx.info/api/pool_stats (only available until next FLUSHALL)
Uploaded TXT file coming soon...

@leshacat leshacat added Bug API Bug Help Wanted labels on 1 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 1 Nov 2018 •
Payment processor is first, this is second, adding algos is third for v1.1.4 - then it will be finalized just to get stability back in the repository since the initial forking.

If anyone wishes to fork this and work on this in the meantime, I would appreciate it :) Just be sure to submit a PR to get credited for work :)

@leshacat leshacat added the Validation Bug label on 1 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 1 Nov 2018
Related to issue https://github.com/foxer666/node-open-mining-portal/issues/106
(worker shares not being validated in payment processor OR api calls)

@leshacat leshacat added Help Wanted v1.1.4 finalization Medium Priority and removed Help Wanted labels on 1 Nov 2018
@leshacat leshacat changed the title Pool/Miner stats stop working: API is not validating workers before retrieving data Pool/Miner stats stop working: API is not validating workers before retrieving data from redis (some redis data is invalid miners) on 2 Nov 2018
@leshacat leshacat added Redis Bug High Priority and removed Medium Priority labels on 2 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 3 Nov 2018 •
Turns out this issue is causing charts to stop showing...

https://github.com/foxer666/node-open-mining-portal/issues/106 does not seem to fix it.

Raised priority, this creates a bad user experience.

This was referenced on 3 Nov 2018
Open
Pool/Miner stats stop working zone117x/node-open-mining-portal#644
Open
Pool/Miner stats stop working SkinnyPeteTheGiraffe#10
@leshacat leshacat added Needs Investigating In Progress Critical Priority labels on 4 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •
This affects pools for ALL coins, not just one. Also note, I have not yet even touched the API codes yet...

Moved to top of priority list, investigating now... Could use help, in murky waters...

With bugs like this, I am surprised NOMP even worked in the first place... I think I may not be from the days of NOMP, I think I am from the days of MPOS - before php-mpos namechange - I musta been really medicated :P

@leshacat leshacat self-assigned this on 4 Nov 2018
@leshacat leshacat removed the High Priority label on 4 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •
Note to self: install redis tools to look at redis data, try to locate the data causing the crash, and remove it. This should restore charts and confirm the issue.

@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •

Turns out I was wrong... But installing RedisMin has brought me closer to the issue.

When I access the "StatHistory" table, it gives an error "ESOCKETOVERFLOW" and the redis server restarts...

Reloading any page with graphs on it, and going back to RedisMin, gives the same error + restart.

Suspecting a corrupted table somehow?

Screenshot attached.

Ideas?

@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018 •
Purging Redis data clears the issue, and at some point it comes back...

Solution: Need actual DB (MySQL/MariaDB) to operate properly (and with less memory)

Flushed RDB on EasyX Pool, charts back, miners back, redis tables / data back, and now I can view that table.

Will keep eye on RDB resume work on this issue when problem returns

@leshacat leshacat removed the In Progress label on 4 Nov 2018
@leshacat leshacat referenced this issue on 4 Nov 2018
Open
Add MySQL / MariaDB / MongoDB option #157
@leshacat
MemberAuthor
leshacat commented on 4 Nov 2018
This could be due to the memory filling up (because we are using a memory cache like a database)

@leshacat leshacat changed the title Pool/Miner stats stop working: API is not validating workers before retrieving data from redis (some redis data is invalid miners) Pool/Miner stats stop working on 4 Nov 2018
@leshacat leshacat added Needs Testing and removed Help Wanted labels on 4 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 5 Nov 2018 •

The above 14MB in memory is from a few hours of mining slowly... If we dump the hashrate we did before, I could see the memory of the server filling up easily. This is only with 4 miners. Imagine running an average sized pool with this...

Awaiting more testing data...

@leshacat
MemberAuthor
leshacat commented on 6 Nov 2018 •
#140 (comment)

More data: It just happened again. I can access all tables except the last one, statHistory.
There must be invalid data causing a BOF when accessing it, because RedisMin can't even access it.

I will try to flush just that table and see what happens.

[UPDATE] redis-cli DEL statHistory and restarting pool clears the issue temporarily, but statistics data is lost.

More reason why we need MySQL

@leshacat leshacat added the GUI / Website Bug label on 7 Nov 2018
@leshacat
MemberAuthor
leshacat commented on 7 Nov 2018 •
Possible fix in config.json (under website):

"historicalRetention": 95040,
Set historicalRetention to 95040 or less. Testing now. Could be related to data storage (redis vs mysql)

@leshacat
MemberAuthor
leshacat commented on 18 Dec 2018 •
Ignore previous comment. It appears this is related to the redis server when accessing statsHistory table.

I suspect corrupted data is making it into the redis DB OR possibly table gets too large and redis gives up.

MySQL replacement (#157) would probably fix this issue.

[mooleshacat]

This issue has been resolved.

The cause was the API code outputting 250 MB of block data that was not even used by the client.

Closing ticket.