xavier242
2022-09-22, 10:52 PM
After some arm twisting, on Thursday our host provider tested each of the harddisks and found one not healthy. They replaced it but said it could take up to 24 hours to rebuild the RAID array. As of late Friday evening (Oct 7 11pm EST) we have not received word that the rebuild was successful. Thus again the forum will be shutdown. If we receive word on Saturday that the rebuild was successful, we'll keep the site up on Sunday. If my hunch is right, that will solve the immediate issues and keep the site alive.
Update 5:05am EST Saturday 10/15/2022:
I received word from U2Lynne that the RAID array rebuild was successful.
Furthermore, my initial research suggested the server didn't have enough RAM. One of the first things I noticed about the server was that there was no free RAM which means the virtual memory system (the swap partition) is keeping the server alive. Not good.
I found an online RAM calculator for MySQL which said we need about 16GB just for the database, not including anything else.
Our host provider was able to double the RAM from 16 to 32GB a few hours ago. We're going to let it run today (Saturday) and see what happens. Cross your fingers.
The specific error we have been seeing is "too many connections" with MariaDB, up to 5000 errors per day. It may be related to the random database corruption issues we've had. For any PHP/MySQL admins, feel free to share your experience. I hope to be able to research the issue in depth tomorrow, like is there a way to queue database queries such that no errors happen no matter how RAM starved a system is? Is MariaDB robust with page faults?
For those who don't know, MariaDB is an open source fork of the MySQL database, and part of what makes TTD work.
Update Monday 10/17/2022:
Fixing the RAID array and the RAM upgrade appears to have solved the immediate hardware issues. We're now only getting a few database "too many connections" errors when making a database backup. They appear to be benign. I think I know what's causing those and may have a solution.
I'm also researching how to optimize the server software settings because I hope to grow the site and attract more traders and seeders.
xavier242
Update 5:05am EST Saturday 10/15/2022:
I received word from U2Lynne that the RAID array rebuild was successful.
Furthermore, my initial research suggested the server didn't have enough RAM. One of the first things I noticed about the server was that there was no free RAM which means the virtual memory system (the swap partition) is keeping the server alive. Not good.
I found an online RAM calculator for MySQL which said we need about 16GB just for the database, not including anything else.
Our host provider was able to double the RAM from 16 to 32GB a few hours ago. We're going to let it run today (Saturday) and see what happens. Cross your fingers.
The specific error we have been seeing is "too many connections" with MariaDB, up to 5000 errors per day. It may be related to the random database corruption issues we've had. For any PHP/MySQL admins, feel free to share your experience. I hope to be able to research the issue in depth tomorrow, like is there a way to queue database queries such that no errors happen no matter how RAM starved a system is? Is MariaDB robust with page faults?
For those who don't know, MariaDB is an open source fork of the MySQL database, and part of what makes TTD work.
Update Monday 10/17/2022:
Fixing the RAID array and the RAM upgrade appears to have solved the immediate hardware issues. We're now only getting a few database "too many connections" errors when making a database backup. They appear to be benign. I think I know what's causing those and may have a solution.
I'm also researching how to optimize the server software settings because I hope to grow the site and attract more traders and seeders.
xavier242