tag:status.recart.com,2005:/historyRecart Status - Incident History2024-03-28T15:47:16ZRecarttag:status.recart.com,2005:Incident/201675022024-03-06T16:52:47Z2024-03-06T16:52:47ZMeta is down<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:16</var> UTC</small><br><strong>Update</strong> - Meta resolved the issues, we did no see any problems with our integrations in the last couple of ours.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>16:25</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>16:25</var> UTC</small><br><strong>Monitoring</strong> - We see incoming requests from Meta services, which means they applied some fixes.<br /><br />We change the status to Monitoring.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>15:33</var> UTC</small><br><strong>Identified</strong> - Meta is down, and Recart's Meta integrations (Ads and Messenger) are not working.<br /><br />Services will be available as soon as Mate fixes its issues.</p>tag:status.recart.com,2005:Incident/199197682024-02-07T10:00:56Z2024-02-07T10:00:56ZPlanned upgrade of kubernetes cluster components<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>08:49</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/195920762024-01-03T10:19:56Z2024-01-03T10:19:56ZPlanned upgrade of kubernetes cluster<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>10:19</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>12:13</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/191314562023-11-16T06:20:44Z2023-11-16T06:20:44ZFacebook Messenger API is down<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>06:20</var> UTC</small><br><strong>Resolved</strong> - The problem has been resolved</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Investigating</strong> - Facebook Messenger API is down, all the API calls result in errors.</p>tag:status.recart.com,2005:Incident/191259892023-11-15T12:32:16Z2023-11-15T12:32:17ZOrder Receipt Follow-Up SMS Messages Not Sending<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>12:32</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>11:38</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the result.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>10:06</var> UTC</small><br><strong>Identified</strong> - We are currently experiencing an issue where order receipt follow-up SMS messages are not being sent as expected. Our technical team is actively investigating the issue to identify the root cause and implement a resolution.</p>tag:status.recart.com,2005:Incident/191111662023-11-13T22:08:42Z2023-11-13T22:08:42ZShopify is down<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>22:08</var> UTC</small><br><strong>Resolved</strong> - The problem has been resolved.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>21:16</var> UTC</small><br><strong>Monitoring</strong> - We see a fix implemented, storefronts are getting back to work.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>21:02</var> UTC</small><br><strong>Identified</strong> - Shopify storefronts are not available at the moment, because of a Shopify outage. This is affecting Recart as well, some part of our system are not capable of working.<br /><br />You can find more info here: https://www.shopifystatus.com/</p>tag:status.recart.com,2005:Incident/186204062023-10-02T11:38:50Z2023-10-02T11:38:50ZOngoing data migration<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>11:38</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>11:15</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>10:11</var> UTC</small><br><strong>Scheduled</strong> - Due to ongoing data migration, cart abandonment metrics are temporarily incorrect</p>tag:status.recart.com,2005:Incident/180182512023-08-02T09:50:44Z2023-08-02T09:50:44ZPlanned upgrade of kubernetes cluster<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>08:39</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>06:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>08:25</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/177608022023-07-05T07:24:12Z2023-07-05T07:24:12ZPlanned upgrade of kubernetes cluster components<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>07:24</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>06:53</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/177530732023-07-04T09:00:21Z2023-07-04T09:00:21ZPlanned upgrade of kubernetes cluster components<p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>07:28</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>06:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>14:34</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/177286112023-07-03T08:15:38Z2023-07-03T08:15:38ZPlanned upgrade of kubernetes cluster<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>08:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>07:40</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>11:12</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/177096792023-06-29T07:54:55Z2023-06-29T07:54:55ZPlanned upgrade of kubernetes cluster components<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>07:54</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>06:39</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>06:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>14:24</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/176958622023-06-27T07:33:53Z2023-06-27T07:33:53ZPlanned upgrade of kubernetes cluster components<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>07:33</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>06:19</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>06:06</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>06:02</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/176307352023-06-21T09:00:25Z2023-06-21T09:00:25ZPlanned upgrade of Elasticache cluster<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>07:46</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>13:20</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/175691372023-06-14T09:04:15Z2023-06-14T09:04:15ZProblems with ghosting process<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Update</strong> - The AWS outage from yesterday caused problems in our system. A few services lost their permission to read from SQS queues.<br /><br />We restarted all the services to make sure every permission-related info is updated and working. <br /><br />All the services are working properly.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>07:50</var> UTC</small><br><strong>Monitoring</strong> - No changes have been released to the affected service before the problem started to happen. We re-deployed the service and it started to operate properly.<br /><br />We are monitoring the results.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>07:33</var> UTC</small><br><strong>Investigating</strong> - We identified a problem with our ghosting subsystem, it started around yesterday 11 PM.</p>tag:status.recart.com,2005:Incident/175592482023-06-14T09:00:04Z2023-06-14T09:00:04ZPlanned upgrade of Elasticache cluster<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>12:39</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/174885142023-06-07T07:58:16Z2023-06-07T07:58:16ZPlanned upgrade of Elasticache cluster<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>07:58</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>11:14</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/174202282023-06-01T15:07:28Z2023-06-01T15:07:28ZPlanned upgrade of kubernetes cluster<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>15:07</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>14:36</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>10:31</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.recart.com,2005:Incident/173282952023-05-23T08:52:37Z2023-05-23T08:52:37ZPlanned upgrade of kubernetes cluster components<p><small>May <var data-var='date'>23</var>, <var data-var='time'>08:52</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>08:07</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>11:39</var> UTC</small><br><strong>Scheduled</strong> - We are upgrading networking components in our kubernetes cluster to newer versions.</p>tag:status.recart.com,2005:Incident/172072972023-05-10T20:28:47Z2023-05-10T20:28:47ZPerformance degradation on our Dashboard<p><small>May <var data-var='date'>10</var>, <var data-var='time'>20:28</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved, our monitors show proper response times.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>19:36</var> UTC</small><br><strong>Monitoring</strong> - The issue has been resolved, performance problems cannot be seen in the system.<br /><br />We are monitoring further to make sure everything is alright.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Update</strong> - The change we applied takes longer than expected, it's around 50% done.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:12</var> UTC</small><br><strong>Update</strong> - We found a resolution to the performance problem. We already started the mitigation. We expect the issue to be resolved in 1-2 hours.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>14:04</var> UTC</small><br><strong>Identified</strong> - We made some optimization to our database clusters. The removal processes started taking more time and generating performance degradation.<br /><br />We are working with our providers on the issue.</p>tag:status.recart.com,2005:Incident/169692462023-04-26T16:55:12Z2023-04-26T16:55:12ZMobile popups does not work properly<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>16:55</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>09:40</var> UTC</small><br><strong>Identified</strong> - We identified the root cause of the error and started to implement a workaround.<br /><br />We are deploying the changes and will start testing in the coming minutes.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>08:22</var> UTC</small><br><strong>Update</strong> - We are still investigating the root cause of the issue.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>07:14</var> UTC</small><br><strong>Update</strong> - The problem is affecting visitors who are using iOS.<br /><br />We are still investigating the root cause, it seems that for half of the visitors is works properly.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>06:43</var> UTC</small><br><strong>Investigating</strong> - We currently investigate why our mobile popups do not work as expected.</p>tag:status.recart.com,2005:Incident/168990642023-04-18T08:04:40Z2023-04-18T08:04:40ZDelays in segmentation<p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>08:04</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>08:03</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>07:36</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue regarding delays in segment creation/updates.</p>tag:status.recart.com,2005:Incident/167500092023-04-04T16:32:46Z2023-04-04T16:32:46ZOur Dashboard is not available in some countries<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Update</strong> - Monitors show us that everything is got back to normal.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:06</var> UTC</small><br><strong>Monitoring</strong> - The issue has been fixed. We are monitoring the DNS propagation right now.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>15:42</var> UTC</small><br><strong>Identified</strong> - We reached out to our provider, we are waiting for their response.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>14:19</var> UTC</small><br><strong>Investigating</strong> - We identified that our Dashboard cannot load in some countries because of an error in one of our provider's networks.</p>tag:status.recart.com,2005:Incident/167003732023-03-30T18:55:46Z2023-03-30T18:55:46ZSegment creation error<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>18:45</var> UTC</small><br><strong>Monitoring</strong> - The fix has been deployed, we are closely monitoring its effect.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>18:32</var> UTC</small><br><strong>Identified</strong> - The root cause has been identified and a fix is getting deployed.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>18:25</var> UTC</small><br><strong>Investigating</strong> - New segments cannot be created in Recart because of an error on our Dashboard. We started the investigation.</p>tag:status.recart.com,2005:Incident/161398422023-02-14T21:35:46Z2023-02-14T21:35:46Zrecart.com is unavailable from the US<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>21:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>21:20</var> UTC</small><br><strong>Monitoring</strong> - A fix has been deployed and our main site is back. We monitor the changes for some time before closing this incident.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>20:38</var> UTC</small><br><strong>Investigating</strong> - We identified that our main website is not available from the US, we started the investigation.</p>