SOAR Backup

SOAR Customers - This page will be updated with new information as soon as it's available. Please check back here as the primary source of communication.

Current Status

1. All SOAR website services are down, including email.

2. We are focusing on getting customers their data and helping them transition to Troopmaster or TroopWebHost for short term needs.

3. Instructions for access customer data has been sent via email. See FAQ.

4. Now working on providing final pieces of data and in more "human friendly" formats.

Updates:

7 Apr 2023 - 10pm ET

Good news, SOAR is coming back! Email with full details sent to customers.

3 Apr 2023 - 10pm ET

soar-export-pages.zip with all pages as individual files and soar-export-announce.zip with all individual announcements files added to customer folders.

2 Apr 2023 - 11pm ET

event-registration.csv with signup data now available for download.

2 Apr 2023 - 8:30pm ET

Changed FAQ to Google Doc to make easier to update.

Customer data is now available! Emails were sent to primary contacts with access instructions. See FAQ for more details.

1 Apr 2023 - 3:30pm ET

Added a FAQ (top right) to help answer common questions. We are focusing on getting data to customers at the moment and don't want to stop to answer individual emails.

1 Apr 2023 - 11am ET

Testing over the last two days has shown that SOAR will not be able to get website services up and running for weeks. Customers need a website today. We have worked with Troopmaster and TroopWebHost to provide a path for you to do that, including transferring your data. Full details can be found in the email communication. Apologies for lack of communication yesterday. It took time to make all these arrangements and do what is best for SOAR customers - getting you a website service immediately!

1 Apr 2023 - 10am ET

Data moved onto new server overnight. Testing data exports, then creating method to provider secure access to customers. Email with more details coming in next couple hours.

31 Mar 2023 - 2:30pm ET

Copy of data from old drives has begun. This is going to take awhile. Terrabytes (TB) of data.

31 Mar 2023 - 1:30pm ET

Progress! New server has been provisioned and they are adding drives from old machine now.

31 Mar 2023 - 10am ET

Lots of work completed yesterday towards the goals of providing customers access to data and plan for restoration of services. Email sent to customers with those details.

30 Mar 2023 - 4pm ET

Make sure you have two unit leaders subscribed to this mailing list so you receive updated information. Patiently waiting for word on access to new server with data drives attached!

30 Mar 2023 - 2am ET

Primary customer contacts (those that renewed last) added to Constant Contact email list and email communication sent with information on this page to make sure all customers are aware of what is going on. Make sure you have two unit leaders subscribed to this mailing list so you receive updated information.

29 Mar 2023 - 7pm ET

New server defined & ordered. Aptum plans to have it online, with customer data drives connected by end of tomorrow. Expecation setting: blank server, updated software versions, going to take some time to get everything working again in new environment. Need to get everything working, then import thousands of customer's data.

29 Mar 2023 - 2pm ET

Decision made to leave the server off to protect the data. Started discussions with Aptum about shorter term getting access to data and then getting new server spun up to be new home.

29 Mar 2023 - noon ET

The act of trying to prepare for migration over the next couple hours taxed the server too much, and it started failing more often.

29 Mar 2023 - 10am ET

Follow up call with Aptum. No replacement motherboards available. Server is end of life and they don't stock them anymore. Bunch of discussions of how that's possible if its being paid for. Focusing on getting a plan to migrate to a new server instead.

29 Mar 2023 - 9am ET

Aptum had their DC data center look at the machine. They noted BIOS errors that could indicate a motherboard issue, and the possible cause of the segment faults. The irony is that these machines have redundant everything (CPU, Memory, RAID Drives, power supplies, network cards, etc) except motherboard. The server was had reset, brought back up, and put back into the previous state described below.

29 Mar 2023 - 5am ET

Rather than continue with the updates, the process of rolling back all the changes was started. By 8am ET all the changes were reverted. The same problem existed though and other issues started showing up. This is when "segment faults" started occurring on the server. In short, these are signs of bigger problems (likely hardware) and they cause the server to reboot mid stream. We called our server provider Aptum at 8am ET to get more information.

At this point SOAR websites were left up and accessible via the SOAR domains. The majority of vanity domains would forward to them, if you bypassed the security warning. Email was still accessible.

28 Mar 2023 - Midnight ET

Routine server maintenance was started. Unexpected errors were encountered during the updates. We started tracking them down, with the belief they were related to the SSL (secure website certificate) issues again. As the night progressed, as one problem was solved, another popped up. We could get the sites up using the SOAR domains (.mytroop.us, mypack.us, mygroup.us) but every time the SSL certificates were added back in for vanity domains (i.e troop123.com) it caused errors.