tag:safetyspectrum.statuspage.io,2005:/historySafetySpectrum Status - Incident History2022-06-30T09:23:10-07:00SafetySpectrumtag:safetyspectrum.statuspage.io,2005:Incident/94008332022-02-24T10:10:20-08:002022-02-24T10:10:20-08:00Connectivity Issues<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:10</var> PST</small><br><strong>Resolved</strong> - This incident is resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:10</var> PST</small><br><strong>Update</strong> - This incident is resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:09</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:03</var> PST</small><br><strong>Update</strong> - Engineers are recovering affected routing components.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>09:37</var> PST</small><br><strong>Investigating</strong> - Engineers are looking into reports of connectivity issues.</p>tag:safetyspectrum.statuspage.io,2005:Incident/36942152020-03-16T09:54:39-07:002021-10-14T12:22:02-07:00Mobile App Outage<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Resolved</strong> - The incident has been resolved. All systems are operational and functioning normally.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:49</var> PDT</small><br><strong>Monitoring</strong> - Mobile and Web Apps now fully operational and functioning normally. We continue to monitor the performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:47</var> PDT</small><br><strong>Update</strong> - The routing to US apps has now stabilized. The situation is being monitored while working to find the root cause of this disruption.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:34</var> PDT</small><br><strong>Update</strong> - We have detected routing issues on the Mobile and Web Apps. We are working to resolve the issue as soon as possible. At this time both Mobile and Web apps are online and functional. However, due to the routing issue some users may experience degraded performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:28</var> PDT</small><br><strong>Update</strong> - Mobile app is online. Some users may experience degraded performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:20</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:safetyspectrum.statuspage.io,2005:Incident/26989032019-07-12T22:00:02-07:002021-10-14T12:22:02-07:00Intermittent SSL Certificate Failures.<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>18:07</var> PDT</small><br><strong>Monitoring</strong> - A certificate caching issue has been identified and corrected. The system performance is now being monitored.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>15:29</var> PDT</small><br><strong>Investigating</strong> - The encrypted connection the KnowledgeKeeper cloud is experiencing intermittent SSL certificate failures. Response times are currently degraded for both the Web and Mobile apps.</p>tag:safetyspectrum.statuspage.io,2005:Incident/23315252019-04-16T10:33:33-07:002021-10-14T12:22:02-07:00Secure HTTPS Connection Errors<p><small>Apr <var data-var='date'>16</var>, <var data-var='time'>10:33</var> PDT</small><br><strong>Resolved</strong> - The certificate error has been resolved and the Web App pages are loading normally again.</p><p><small>Apr <var data-var='date'>16</var>, <var data-var='time'>08:40</var> PDT</small><br><strong>Identified</strong> - We have identified a certificate error on the Web App.</p><p><small>Apr <var data-var='date'>16</var>, <var data-var='time'>08:09</var> PDT</small><br><strong>Investigating</strong> - When accessing Knowledgekeeper via the Web App the browser may give a warning that connection is not secure.</p>tag:safetyspectrum.statuspage.io,2005:Incident/22130152019-02-18T13:52:03-08:002021-10-14T12:22:02-07:00Elevated API Errors<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:52</var> PST</small><br><strong>Resolved</strong> - A push certificate issue was discovered and corrected. This incident has been resolved. The mobile app and web app have been restored to normal operation.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:17</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>13:17</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:safetyspectrum.statuspage.io,2005:Incident/16126822018-02-21T18:14:17-08:002021-10-14T12:22:02-07:00Web App Offline<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>18:14</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. The Web App has been restored to full funtionality.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>17:42</var> PST</small><br><strong>Identified</strong> - We have identified the root cause of the Web App issue and expect to resolve ti shortly.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>17:20</var> PST</small><br><strong>Investigating</strong> - The web app has experienced an event that has taken it offline. We are investing the cause.</p>tag:safetyspectrum.statuspage.io,2005:Incident/15678362018-01-16T16:21:47-08:002021-10-14T12:22:02-07:00Data Processing Delays - Mobile and Web Apps are Affected<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>16:21</var> PST</small><br><strong>Resolved</strong> - This issue has been resolved.
<br />The mobile and web apps are now functioning normally.
<br />
<br />We apologize for any inconvenience this may of have caused.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>16:07</var> PST</small><br><strong>Monitoring</strong> - We believe that we have resolved this issue.
<br />The mobile and web apps are now functioning normally.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>16:02</var> PST</small><br><strong>Identified</strong> - The data processing issue has been identified. Service will return to normal shortly.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>15:31</var> PST</small><br><strong>Investigating</strong> - Our data processing infrastructure is not responding to requests. We are investigating the issue.</p>tag:safetyspectrum.statuspage.io,2005:Incident/14433442017-10-25T15:00:47-07:002021-10-14T12:22:03-07:00KnowledgeKeeper Cloud Function Issue: Invite New Members<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - The issue has been resolved. The invitation workflow is now working normally.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>13:54</var> PDT</small><br><strong>Monitoring</strong> - This issue is cleared and we are monitoring and testing to be sure it is resolved.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>07:24</var> PDT</small><br><strong>Investigating</strong> - Currently there is an issue in the KnowledgeKeeper Cloud that is causing an error when inviting new members. This affects inviting new members and sending invitations to existing members or groups of members to watch a video journal.</p>tag:safetyspectrum.statuspage.io,2005:Incident/12888582017-07-09T12:38:45-07:002021-10-14T12:22:03-07:00Unable to Reach Web App<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>12:38</var> PDT</small><br><strong>Resolved</strong> - KnowledgeKeeper Web App SSL certificate is now resolving normally and the Web App is fully operational.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>11:35</var> PDT</small><br><strong>Update</strong> - The DNS for the updated SSL security certificate is propagating now.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>11:10</var> PDT</small><br><strong>Identified</strong> - We are making a DNS change to our SSL security. This should be resolved in the next hour.</p>tag:safetyspectrum.statuspage.io,2005:Incident/12704332017-06-19T12:00:41-07:002021-10-14T12:22:03-07:00CDN Database Issue Affecting Web App Playback<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Resolved</strong> - This CDN issue has been resolved. Video journals are now playing back normally on the Web App.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:19</var> PDT</small><br><strong>Identified</strong> - The CDN Issue has been identified.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:14</var> PDT</small><br><strong>Investigating</strong> - The CDN is experiencing a database issue that is affecting Playback on the Web App.</p>tag:safetyspectrum.statuspage.io,2005:Incident/11560842017-03-12T18:35:40-07:002021-10-14T12:22:03-07:00Video Journal Upload Errors<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>18:35</var> PDT</small><br><strong>Resolved</strong> - The issue with uploading new video journals has been identified and resolved. Upload operations are all running normally.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>15:09</var> PDT</small><br><strong>Update</strong> - We are still investigating and have not determined the cause of this issue yet.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>12:03</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with the Content Development Network (CDN) that is affecting the ability to upload new video journals.</p>tag:safetyspectrum.statuspage.io,2005:Incident/10438722016-12-23T21:50:01-08:002021-10-14T12:22:03-07:00Linked CoPs Switching Issue<p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>21:50</var> PST</small><br><strong>Resolved</strong> - This issue appears to be a DNS issue due to an IP address change when the new virtui machine was loaded. Over the next 24 hours the DNS servers should resolve the newIP fast enough to keep up with the speed of TouchID when switching CoPs.
<br />
<br />Additionally the next iOS app update will do a better job of using TouchID synchronously during CoP switching.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>09:59</var> PST</small><br><strong>Update</strong> - This issue appears to be only affecting CoP switching when TouchID is used. Still investigating.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>07:30</var> PST</small><br><strong>Investigating</strong> - At 2:08 EST one of our virtual machines hosted in AWS_us-east-1) suddenly became unreachable:
<br />
<br />The system failed over to the redundant secondary while the VM was being restarted.
<br />
<br />This process has caused an issue where switching between Linked CoPs causes a session error, and logs the person out.
<br />
<br />We are investigating why this is occurring.</p>tag:safetyspectrum.statuspage.io,2005:Incident/4923892016-05-04T11:57:25-07:002021-10-14T12:22:03-07:00Elevated Error Rates<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>11:57</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>09:01</var> PDT</small><br><strong>Investigating</strong> - We are experiencing elevated error rates in our mobile application.</p>tag:safetyspectrum.statuspage.io,2005:Incident/4908332016-05-03T15:24:51-07:002021-10-14T12:22:03-07:00DNS Errors<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>15:24</var> PDT</small><br><strong>Resolved</strong> - The DNS routing has resolved and performance of the Web App has returned to normal.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>12:34</var> PDT</small><br><strong>Monitoring</strong> - We have been updating our DNS names to prepare for Apple Universal links. While the DNS names are propagating across the internet there are some intermittent connection issues to the Web App.</p>tag:safetyspectrum.statuspage.io,2005:Incident/4736482016-04-21T15:00:04-07:002021-10-14T12:22:03-07:00Elevated Timeout Errors<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>14:11</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of timeout errors and are currently looking into the issue.</p>tag:safetyspectrum.statuspage.io,2005:Incident/3066722015-09-02T10:44:57-07:002021-10-14T12:22:03-07:00Timeout Errors<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>10:44</var> PDT</small><br><strong>Resolved</strong> - The issue causing the timeout errors has been resolved. The web application response time has return to normal limits of operation.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>07:11</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of timeout errors and are currently looking into the issue.</p>tag:safetyspectrum.statuspage.io,2005:Incident/2759452015-07-06T17:44:24-07:002021-10-14T12:22:03-07:00Elevated Timeout Errors<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>17:44</var> PDT</small><br><strong>Resolved</strong> - From 14:15 to 16:30 PST today the Website App was experiencing degraded performance. During this time frame there were high levels of timeout errors. While timeout errors do randomly occur for various reasons, the frequency of the errors was very high with at least one timeout error every 5-10 minutes during a two hour time frame.
<br />
<br />The root cause has not been identified yet but the website app has been restored to its normal, high performance operating state.</p>tag:safetyspectrum.statuspage.io,2005:Incident/1904532015-03-23T14:02:42-07:002021-10-14T12:22:03-07:00DNS Errors<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Resolved</strong> - The Web Application is now working again at full performance from all geographic locations with no timeouts.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>12:35</var> PDT</small><br><strong>Identified</strong> - We believe we have identified the problem and it has to do with broadband access in the Eastern Part of the U.S. Our monitoring servers in Atlanta, GA and Philadelphia, PA and in Europe have been experiencing timeout errors when trying to access the KnowledgeKeeper Web Application.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:36</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of domain name server (DNS) or redirect errors and are currently looking into the issue.</p>