tag:funraise.statuspage.io,2005:/historyFunraise Status - Incident History2024-03-27T19:50:18-07:00Funraisetag:funraise.statuspage.io,2005:Incident/182495022023-08-24T13:00:19-07:002023-08-24T21:28:34-07:00Paypal Buttons Disappearing<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>13:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:33</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:33</var> PDT</small><br><strong>Monitoring</strong> - It has been reported by users and verified by our internal team that the PayPal buttons are intermittently disappearing from donation forms. Users affected by this issue are temporarily unable to make transactions through PayPal while the buttons are not visible.<br /><br />Preliminary investigations indicate that the root cause of this issue resides on PayPal's end. PayPal is working on tracking the resolution on their incident page: https://www.paypal-status.com/incident/production</p>tag:funraise.statuspage.io,2005:Incident/179038232023-07-18T15:00:00-07:002023-07-19T12:46:14-07:00Upstream Payment Provider Intermittent Errors<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - Since 2023-07-18 3:01 ET, there was an issue detected with an upstream payments processor causing intermittent donation failures for a subset of customers for a subset of donations. The upstream provider resolved the issue as of 20:00 ET.</p>tag:funraise.statuspage.io,2005:Incident/157455622022-12-30T18:41:47-08:002022-12-30T18:41:47-08:00Some PayPal Donations Are Failing For Legacy PayPal<p><small>Dec <var data-var='date'>30</var>, <var data-var='time'>18:41</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>30</var>, <var data-var='time'>16:56</var> PST</small><br><strong>Update</strong> - We are continuing to monitor this issue with our upstream payments provider. We're seeing intermittent failures with other payment methods.</p><p><small>Dec <var data-var='date'>30</var>, <var data-var='time'>13:34</var> PST</small><br><strong>Update</strong> - We are continuing to monitor this issue.</p><p><small>Dec <var data-var='date'>30</var>, <var data-var='time'>13:34</var> PST</small><br><strong>Identified</strong> - We've identified an issue with an upstream payments provider (Legacy Delegate Mode PayPal) which is causing intermittent failures for a small number of customers and a subset of PayPal donations. We are currently monitoring the upstream provider for restoration of service.</p>tag:funraise.statuspage.io,2005:Incident/104741442022-07-05T13:07:25-07:002022-07-05T13:08:42-07:00Performance Degredation<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>13:07</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>13:04</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>13:02</var> PDT</small><br><strong>Investigating</strong> - We've identified an underlying hardware fault at our data tier that is causing some requests to slow down. We're working to add capacity to restore performance.</p>tag:funraise.statuspage.io,2005:Incident/103652792022-06-21T12:18:26-07:002022-06-21T12:18:26-07:00Platform Outage - Upstream Service Provider<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>12:18</var> PDT</small><br><strong>Resolved</strong> - This incident is resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:03</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>00:17</var> PDT</small><br><strong>Identified</strong> - We have identified an issue with our upstream hosting service.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>23:18</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating a system wide outage due to an upstream service provider.</p>tag:funraise.statuspage.io,2005:Incident/100521772022-05-22T20:54:46-07:002022-05-22T20:54:46-07:002022.5.0<p><small>May <var data-var='date'>22</var>, <var data-var='time'>20:54</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>20:45</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>20:22</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>19:15</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/98075892022-04-18T17:35:50-07:002022-04-18T17:35:50-07:002022.4.0 Release Window<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:35</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>17:21</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>16:55</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/96417732022-03-27T08:31:47-07:002022-03-27T08:31:47-07:002022.3.0<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>08:31</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>08:30</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>08:24</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/94300592022-03-02T11:11:20-08:002022-03-02T11:11:20-08:00Not All Recent Donations Are Visible In Platform<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>11:11</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>20:00</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and is being rolled out across Funraise accounts.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:08</var> PST</small><br><strong>Identified</strong> - We're currently investigating reports of some platform accounts not showing fully up-to-date donation data. Account data is still being stored and ingested and donations are processing correctly but just are not showing up correctly in lists at the moment. We're working on resolving this issue and expect no data loss.</p>tag:funraise.statuspage.io,2005:Incident/89144252021-12-22T04:00:00-08:002021-12-22T09:27:59-08:00Datacenter Power Outage Causes Failed Donations<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>04:00</var> PST</small><br><strong>Resolved</strong> - A handful (~10) of donations using our new PayPal integration failed to process between 4:15am to 6:30am Pacific Standard Time due to a power loss at an Amazon data center. Amazon resolved the situation and donations are processing normally through that integration since 6:30am PST.</p>tag:funraise.statuspage.io,2005:Incident/88968582021-12-12T23:00:00-08:002021-12-20T09:46:21-08:00Log4j Statement<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>23:00</var> PST</small><br><strong>Resolved</strong> - Funraise for the does not use log4j for any of its native application code. Funraise identified one area of log4j use in a 3rd party component that we use for search engine functionality (ElasticSearch) that is hosted separately from the Funraise Platform core infrastructure. Funraise applied the appropriate initial patches and mitigation strategies on Dec 13 and continues to monitor the situation as new recommendations and patches are released and has found no evidence of exploit or attempted exploit in our application logs.<br /><br />Log4j was also identified in a 3rd party component powering our BI functionality. This functionality is also hosted separately from the Funraise core infrastructure and presents lowered risk of RCE. The vendor for that component patched and updated their infrastructure on Dec 16.<br /><br />Funraise continues to maintain a proactive security posture and undergoes annual analysis of our codebase and infrastructure by an independent security firm.</p>tag:funraise.statuspage.io,2005:Incident/83634152021-10-31T10:04:17-07:002021-10-31T10:04:17-07:002021.9.0 Release Window<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>10:04</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>10:02</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>09:52</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>09:18</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/80322212021-09-19T19:37:48-07:002021-09-19T19:37:48-07:002021.9.0 Release Window<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>19:37</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>19:28</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>18:45</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/78957952021-09-02T20:04:57-07:002021-09-02T20:04:58-07:002021.8.0 Release Window<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>20:04</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>18:44</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>15:50</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/79412132021-08-31T17:30:00-07:002021-09-07T16:39:59-07:00Platform Unavailability<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>17:30</var> PDT</small><br><strong>Resolved</strong> - The Funraise platform suffered a downtime from 1:31:11am UTC - 2:05:07am UTC due to insufficient compute to spawn new application instances. Funraise has increased server capacity to mitigate this issue going forward.</p>tag:funraise.statuspage.io,2005:Incident/78499062021-08-28T10:45:30-07:002021-08-28T10:45:30-07:00Planned Downtime<p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>10:45</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>10:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>10:30</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/76184242021-07-27T15:30:04-07:002021-07-27T19:09:39-07:00Lists Showing "No Data" In Certain Situations<p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>15:01</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Investigating</strong> - We're currently investigating reports of some platform accounts showing no data in certain sections. Account data is still being stored and ingested and donations are processing correctly but just are not showing up correctly in lists at the moment. We're working on resolving this issue and expect no data loss.</p>tag:funraise.statuspage.io,2005:Incident/76083182021-07-26T11:00:00-07:002021-07-26T11:50:28-07:00Platform Downtime<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Resolved</strong> - Funraise platform suffered a downtime between 11:06am-11:17am PDT. The cause was underlying infrastructure not being able to boot a new set of application servers in a timely manner. Our systems team responded quickly and restarted our application server formation which provisioned a new set of servers which started successfully.</p>tag:funraise.statuspage.io,2005:Incident/73458482021-06-27T11:00:30-07:002021-06-27T11:00:30-07:002021.6.0 Release Window<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>10:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>09:32</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/72379542021-06-13T10:47:09-07:002021-06-13T10:47:09-07:00Database Maintenance Window<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>10:47</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>09:40</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>09:15</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/70057592021-05-19T19:00:58-07:002021-05-19T19:00:58-07:002021.5.0 Release Window<p><small>May <var data-var='date'>19</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>18:38</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>18:19</var> PDT</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>18:18</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>08:49</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/67804552021-04-18T10:37:43-07:002021-04-18T10:37:43-07:002021.4.0 Release Window<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>10:37</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>10:34</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>10:31</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>10:18</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>10:09</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/66238122021-03-27T11:18:09-07:002021-03-27T11:18:09-07:002021.3.0 Release Window<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>11:18</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>11:15</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>11:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>10:58</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/64206132021-02-28T10:10:27-08:002021-02-28T10:10:27-08:002021.2.0 Release Window<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:10</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>10:03</var> PST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:57</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:02</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:funraise.statuspage.io,2005:Incident/62149072021-02-07T14:20:29-08:002021-02-07T14:20:29-08:002021.1.0 Release Window<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:20</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:15</var> PST</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>12:55</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>