Cristobal M
11/6/23, 10:47:59 AM EST
OK, about AGBU what I discovered Friday (and did some little research on Saturday) there are like 20 assets that are 132kb original size and modification date is 1970. Hard to track why this happen but I found that last week Rundek (Ansible-docker) failed to propagate certificates in some machines, one of those is m38, that hosts Agbu-B (it is old Agbu-a really but we switched them), I was looking the original assets there but they are not in the B site, Sync failed since the SSL Cert not propagated.
Then, I restarted Rundek docker because it was not responding and now I can't put it back on line, I manually moved the Certs to m38 but I'm still trying to restore Rundek.
About the Agbu corrupted files not sure what to do, I also reviewed the Backupp machine and I dont see them there.
Dieneke S
11/7/23, 10:41:29 AM EST
https://zoom.us/j/94987892974?pwd=S3VSZ3M1MEVvQXhNRFU2NWxMY055QT09
My bad, yuou guys can use this link
My bad, yuou guys can use this link
Christopher B
11/7/23, 3:20:54 PM EST
Cristobal M
11/8/23, 12:10:49 PM EST
I have bad news, after Rsync in AGBU we still have 6589 corrupted files (only 100 files were restored from old site)
Cristobal M
11/8/23, 12:14:24 PM EST
Next thing I'm doing is: gonna backup Old Site assets, restore from Backuppc there and see if we can recover some more.
Cristobal M
11/8/23, 1:18:10 PM EST
Sadly, I think we are not gonna find too much, I searched several random-photos in backuppc and they are not there. Some of the files have "2022" in the filename I was expecting some of those will appear, but that does not necessarily means they were uploaded on 2022.
I still can't identify a common thing in missing files, they are very random paths and very different upload dates.
Process I described early still running (gonna restore what we have in backuppc, I'll notify what I found when is done)
Cristobal M
11/8/23, 2:04:32 PM EST
It says meeting room is bussy??
any ways I need 10 minutes to pickup something couple blocks from here, brb
Cristobal M
11/8/23, 2:42:00 PM EST
I was able to reproduce the Sync issue, and I think I found 2 factors make it happen. If you a re avail I can show it to you
Cristobal M
11/8/23, 4:07:09 PM EST
Ok, it is this way:
-The "Push originals" option it was not enabled in the B site (the one it became the Prod server), so any new file uploaded to this node was Synced to the A-site but the original was not sent.
-The 2nd part of the issue is that some how, Assets are being "updated" in the A-site and then synced-back to B site, now B site tries to download the original and fails, the original is not in the A site, but anyways writes the response to the hard drive overwriting the original file.
Christopher B
11/8/23, 5:09:11 PM EST
Christopher B
11/9/23, 1:00:43 PM EST
Cristobal M
11/9/23, 5:57:59 PM EST
This is the doc
>https://docs.google.com/spreadsheets/d/1MgdK8KGXU_lWtmPNSmFaYxANy-ZjRwv1w810OeTN4pU/edit?usp=sharing
>https://docs.google.com/spreadsheets/d/1MgdK8KGXU_lWtmPNSmFaYxANy-ZjRwv1w810OeTN4pU/edit?usp=sharing
Cristobal M
11/10/23, 11:08:46 AM EST
Found The Music Academy SSL expired, I need to finish Ansible config