tag:status.faithlife.com,2005:/historyFaithlife Status - Incident History2024-03-28T05:51:25-07:00Faithlifetag:status.faithlife.com,2005:Incident/203070682024-03-19T22:24:01-07:002024-03-19T22:24:02-07:00Logos.com is down due to database server maintenance<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>22:24</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>22:05</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>21:35</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with our services going down due to database maintenance. We will keep this page updated as we continue to learn more and work to resolve the issue.</p>tag:status.faithlife.com,2005:Incident/194666132023-12-17T18:40:13-08:002023-12-17T18:40:13-08:00Core services are impacted by a certificate issue<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>18:40</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>18:35</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>18:23</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>18:21</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.faithlife.com,2005:Incident/186052782023-10-03T05:38:34-07:002023-10-03T05:38:34-07:00Server Maintenance<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>05:38</var> PDT</small><br><strong>Completed</strong> - The maintenance is complete. We are continuing to monitor our systems. Thank you for your patience.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:35</var> PDT</small><br><strong>Update</strong> - Maintenance has been extended by 2 hours to 6 AM Pacific. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>12:15</var> PDT</small><br><strong>Update</strong> - We will be closely monitoring affected services for an additional 24 hours for any performance impact.</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>15:08</var> PDT</small><br><strong>Scheduled</strong> - On October 3rd, starting at 1 AM Pacific, we will be performing server maintenance. We expect any work that may cause an outage for any of our sites to be completed within 3 hours. During this period websites may be intermittently unavailable.<br /><br />The following sites may be affected during this period.<br /><br />Account Sign In<br />App.logos.com and Learning Tools Integration<br />Sync<br />Proclaim Services<br />Faithlife.com</p>tag:status.faithlife.com,2005:Incident/178812322023-07-17T15:20:42-07:002023-07-17T15:20:42-07:00Logos.com is unavailable<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>15:20</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:49</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:48</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:46</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:43</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:42</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:07</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>13:50</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>12:02</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/178550092023-07-14T08:48:05-07:002023-07-14T08:48:06-07:00E-Commerce Sites are Down Due to Maintenance<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>08:48</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>07:52</var> PDT</small><br><strong>Identified</strong> - A standard database maintenance procedure is taking longer than expected.</p>tag:status.faithlife.com,2005:Incident/176104642023-06-18T09:00:47-07:002023-06-18T09:00:47-07:00Elevated Locations API Errors<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>07:14</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>06:55</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of Locations API errors and are currently looking into the issue.</p>tag:status.faithlife.com,2005:Incident/175005552023-06-07T09:48:40-07:002023-06-07T09:48:41-07:00Database failure<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>09:48</var> PDT</small><br><strong>Resolved</strong> - Service is restored.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>05:07</var> PDT</small><br><strong>Identified</strong> - We have identified a database failure that will cause widespread outages today. We're working on it!</p>tag:status.faithlife.com,2005:Incident/174786292023-06-05T10:29:55-07:002023-06-05T10:29:55-07:00Product search and access is slow/failing<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>10:29</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>10:29</var> PDT</small><br><strong>Update</strong> - All systems are back to normal. Thank you for your patience!</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>09:52</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>09:32</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>09:32</var> PDT</small><br><strong>Update</strong> - We are temporarily and briefly taking some application capabilities offline in order to put in a better fix. Should be back online in a few minutes.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>07:59</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>07:17</var> PDT</small><br><strong>Update</strong> - We've found the root cause and added workarounds for the underlying issue. All applications and websites should now be operational, although several will have degraded performance until the underlying issue is fixed.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>07:00</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>06:46</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>06:07</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/171113922023-05-01T11:11:04-07:002023-05-01T11:11:05-07:00Logos Stores are offline<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>11:11</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>10:47</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>10:35</var> PDT</small><br><strong>Investigating</strong> - Logos ecommerce websites, including logos.com and verbum.com, are currently inaccessible.</p>tag:status.faithlife.com,2005:Incident/169733122023-04-26T16:06:55-07:002023-04-26T16:06:55-07:00Expired Certificate for signage.faithlife.com<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>16:06</var> PDT</small><br><strong>Resolved</strong> - An updated certificate has been installed.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>09:17</var> PDT</small><br><strong>Identified</strong> - The Issue has been identified and we are working on implementing a fix</p>tag:status.faithlife.com,2005:Incident/162666782023-02-24T05:46:35-08:002023-02-24T05:46:35-08:00Faithlife.com is currently down<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:46</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:32</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:05</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:04</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>04:55</var> PST</small><br><strong>Investigating</strong> - Faithlife.com is currently down and we are actively investigating this issue.</p>tag:status.faithlife.com,2005:Incident/162399512023-02-22T13:54:56-08:002023-02-22T13:54:56-08:00Unable to Load Church Sites<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>13:54</var> PST</small><br><strong>Resolved</strong> - We have identified the issued and deployed a fix. We will continue to monitor but as of now church sites will load and continue to operate as normal.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>13:45</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/160108662023-02-02T17:10:09-08:002023-02-02T17:10:09-08:00Degraded payment processing performance<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>17:10</var> PST</small><br><strong>Resolved</strong> - The issue with our downstream payment processor has been resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:19</var> PST</small><br><strong>Identified</strong> - We have identified the issue as a connectivity problem with our downstream payment processing services.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:14</var> PST</small><br><strong>Investigating</strong> - We are currently investigating delayed order payment processing. We will keep this page updated as we learn more and work to resolve the issue.</p>tag:status.faithlife.com,2005:Incident/153666762022-12-19T08:30:36-08:002022-12-19T08:30:37-08:00Unable to publish scheduled post<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>08:30</var> PST</small><br><strong>Resolved</strong> - All affected posts have been published and newly-scheduled posts continue to go out on schedule.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>16:39</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>15:39</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue. This only impacts scheduled posts and all other posts are working as expected. We'll provide updates as they come.</p>tag:status.faithlife.com,2005:Incident/152419652022-12-15T04:34:15-08:002022-12-15T04:34:16-08:00ProductsApi Outage<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>04:34</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>03:38</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>03:00</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/130929672022-11-15T21:00:42-08:002022-11-15T21:00:43-08:00Planned network maintenance<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>21:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>20:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>16:54</var> PST</small><br><strong>Scheduled</strong> - We'll be performing network maintenance that will disrupt a few Faithlife services for a short period. We expect the downtime to last about 15 minutes. This will affect both Ministry Tracker and Faithlife Sites.</p>tag:status.faithlife.com,2005:Incident/127012542022-11-02T21:52:18-07:002022-11-02T21:52:18-07:00Partial service disruption on multiple sites.<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>21:52</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>21:45</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>20:28</var> PDT</small><br><strong>Monitoring</strong> - One of our cloud service providers is experiencing a networking issue which is preventing our apps from functioning normally.</p>tag:status.faithlife.com,2005:Incident/126801282022-11-01T09:59:32-07:002022-11-01T09:59:33-07:00Payment Processor Outage<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>09:59</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>08:08</var> PDT</small><br><strong>Update</strong> - We are able to process payments and create billing profiles, we’ll continue to monitor.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>00:45</var> PDT</small><br><strong>Monitoring</strong> - We're awaiting a resolution from our payment processor.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>00:44</var> PDT</small><br><strong>Investigating</strong> - We are currently unable to process payments or create billing profiles due to an outage with our payment processor.</p>tag:status.faithlife.com,2005:Incident/126662472022-10-30T18:20:40-07:002022-10-30T18:20:40-07:00Multiple site outages<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>18:20</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>15:25</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/123865762022-10-20T12:57:49-07:002022-10-20T12:57:49-07:00Logos for iOS/Android stuck loading library data<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>12:57</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>12:29</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>11:21</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.faithlife.com,2005:Incident/119452782022-10-13T03:00:06-07:002022-10-13T03:00:07-07:00Potential issues surrounding Logos/Proclaim updates<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>03:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>02:47</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>02:07</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating possible issues surrounding updating to newer versions of Logos and Proclaim.</p>tag:status.faithlife.com,2005:Incident/112426902022-09-26T10:55:26-07:002022-09-26T10:55:26-07:00Sync Maintenance - Canvas and Bibliography Documents<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>10:55</var> PDT</small><br><strong>Resolved</strong> - Canvas and Bibliography documents are able to sync again.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>20:10</var> PDT</small><br><strong>Investigating</strong> - Our Canvas and Bibliography document sync services will be undergoing a temporary maintenance. All other document types will continue syncing normally. You may see a red exclamation point indicating a sync failure during this maintenance window.</p>tag:status.faithlife.com,2005:Incident/112342892022-09-23T06:43:34-07:002022-09-23T06:43:35-07:00Web app failing to load resources<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>06:43</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>06:15</var> PDT</small><br><strong>Investigating</strong> - Some users are experiencing failure to load resources in the Logos web app. We are currently investigating the issue.</p>tag:status.faithlife.com,2005:Incident/108760002022-08-22T09:44:36-07:002022-08-22T09:44:37-07:00Faithlife Live Stream connection issues<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>09:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating possible connection errors to the RTMP server.</p>tag:status.faithlife.com,2005:Incident/108409882022-08-16T23:39:18-07:002022-08-16T23:39:18-07:00Monthly Server Maintenance<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>23:39</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>15:16</var> PDT</small><br><strong>Scheduled</strong> - On August 16th, starting at 10 PM Pacific, we will be performing our monthly server and network maintenance (this maintenance is always scheduled for the third Tuesday of the month). We expect any work that may cause an outage for any of our sites to be completed by midnight.<br /><br />The following sites may be affected during this period.<br /><br />Logos.com, Verbum.com, and all other e-commerce sites<br />App.logos.com and Learning Tools Integration<br />Licenses APIs<br />Resources APIs<br />Products API</p>