Basic Cloud Services (OTC) Identity Management and Login (OTC) Process Engine (OTC) Viewing (OTC) Import (OTC) Search (OTC) AI Services (OTC) Administration (OTC) ERP Integrations (OTC) d.velop sign (OTC) Contracts (OTC) Administration (OTC) Search (OTC) Administration (OTC) Center (OTC) Outlook Integration (OTC) Embedded View (OTC) Search (OTC) d.velop contracts (OTC)
(06. Jul, 18:25) Resolved - The issue has been successfully resolved by the ISP. The affected peering connection was restored at 04:15 PM. Since then, we have closely monitored all relevant systems and, as of 06:00 PM, have not observed any further errors or service degradation. We have therefore returned to normal operating conditions and consider the incident resolved. We have requested a Root Cause Analysis (RCA) from T‑Cloud Public to better understand the underlying cause and will review the findings once available. Thank you to everyone involved in the investigation and resolution of this issue. We also sincerely thank our customers for their patience and understanding throughout the incident. Have a great evening!
(06. Jul, 15:37) Disrupted - The ISP has identified a potential issue and plans to re-enable the affected peering connection in a controlled manner today at 04:15 PM. Together with T‑Cloud Public, we will closely monitor all relevant systems and error rates during and after the change. If we observe any signs of increased error rates or service degradation, the change will be rolled back immediately. The maximum test window is planned to be 30 minutes. At the moment, the affected peering remains disabled and we are not observing any customer impact. We will provide an update after the test has been completed or earlier if significant findings arise.
(06. Jul, 11:47) Disrupted - The ISP is continuing its investigation into the underlying peering issue. As previously communicated, the affected peering connection remains disabled. As a result, we are currently not observing any service degradation, and all d.velop products are operating as expected. At this time, there are no indications of any customer impact. We remain in close contact with T‑Cloud Public and will provide further updates as soon as new information becomes available.
(03. Jul, 13:02) Disrupted - The ISP is continuing its investigation, and T‑Cloud Public has further escalated the issue in order to achieve a timely resolution. For the next few days, including the upcoming weekend, the affected peering connection will remain disabled. Based on the observations since disabling this routing path, no service degradation is expected for our products, and customers should be able to use them as usual. At this time, we have no indication of any ongoing issues and continue to observe normal system behaviour. We will provide updates if the situation changes, including over the weekend. Otherwise, the next update will be published here on Monday. Thank you for your patience and understanding.
(02. Jul, 17:07) Disrupted - The responsible ISP temporarily re-enabled the affected peering connection in a controlled manner at 04:22:48 PM. Representatives from T‑Cloud Public and d.velop participated in and closely monitored the process. Shortly after the routing path was brought back online, we observed an increase in error rates across affected d.velop products. The ISP immediately resumed its analysis of the routing behaviour. To prevent further customer impact, the routing path was disabled again at 05:00:50 PM. Since the routing path was disabled, we have not observed any further errors and systems continue to operate with expected performance. The ISP will continue its investigation to identify the root cause and determine a permanent solution. Unfortunately, a final resolution is not yet in sight. At the moment, customers should not experience any issues. We will provide further updates as soon as more information becomes available, or by tomorrow morning at the latest. Thank you for your patience, and we apologise for the continued disruption.
(02. Jul, 11:40) Disrupted - The current situation remains unchanged: we are not seeing any further issues, and one of the routing paths towards T‑Cloud Public remains disabled. While this is encouraging, the investigation is still ongoing. The ISP responsible for the affected peering connection has not yet identified a root cause. T‑Cloud Public and their partners therefore continue to investigate the issue. We will provide further updates as soon as more information becomes available, or by this evening at the latest. Thank you for your patience.
(01. Jul, 16:01) Disrupted - T‑Cloud Public has made further adjustments to the routing between AWS and T‑Cloud Public. Since these changes were implemented, T‑Cloud Public and multiple affected customers, including us, have not observed any further errors. Please note that this is not a resolution notice. T‑Cloud Public is continuing to investigate the underlying issue together with their ISP to identify the root cause and ensure a permanent resolution. At the moment, we are seeing the expected system behaviour and normal response times. We will provide further updates as soon as more information becomes available, or by tomorrow morning at the latest. Thank you for your patience, and we apologise for any disruption this issue may have caused.
(01. Jul, 11:05) Disrupted - A joint sync with T‑Cloud Public took place this morning. Based on their analysis, fully disabling the affected routing paths could have a broader impact on all T‑Cloud Public customers. As a result, T‑Cloud Public is currently evaluating alternative mitigation options. A complete traffic rerouting is considered the last option, as it could lead to a full service disruption rather than the currently observed performance degradation. We aligned on this approach, as a potential full outage would have a significantly higher customer impact than the current situation. We continue to monitor the situation closely and will provide further updates as soon as new information becomes available.
(01. Jul, 06:44) Disrupted - The involved ISP has completed initial testing in the night and provided their results to T‑Cloud Public. Based on this, T‑Cloud Public is currently evaluating and discussing the next steps internally. We continue to monitor the situation closely and will provide further updates as soon as new information becomes available.
(30. Jun, 22:16) Disrupted - We are still awaiting further investigation results from T‑Cloud Public and their involved third‑party providers. At this point, no significant new information is available. We continue to closely monitor the situation. We will provide the next update as soon as new information becomes available, or by tomorrow morning at the latest.
(30. Jun, 16:58) Disrupted - T‑Cloud Public has confirmed a peering issue between AWS and T‑Cloud Public as the root cause of the observed latency and timeouts. To mitigate the issue, T‑Cloud Public is planning to disable all routing paths via the affected internet service provider. As this change impacts all T‑Cloud Public customers, it requires careful planning and execution. The current goal is to complete this switch within the next 24 hours. We continue to closely monitor the situation and are awaiting further updates from T‑Cloud Public. We will provide the next update by tomorrow morning at the latest, or earlier if significant new information becomes available.
(30. Jun, 08:44) Disrupted - There is no significant new update at this time. We continue to closely monitor the customer impact and still observe that only a small percentage of requests are affected. As previously communicated, the impact is limited to scenarios where d.velop products hosted on T‑Cloud Public are used in combination with d.velop products hosted on AWS. We are currently awaiting further feedback from T‑Cloud Public and their involved third-party providers. We will provide further updates as soon as new information becomes available.
(29. Jun, 16:19) Disrupted - The issue has been further isolated to a peering problem between T‑Cloud Public and AWS. T‑Cloud Public has involved their internet service provider for further investigation. At the moment, we are dependent on updates from third-party providers. We will share the next update by tomorrow 12:00 PM at the latest, or earlier if significant new information becomes available.
(29. Jun, 13:08) Disrupted - We are continuing to investigate the issue together with T‑Cloud Public (OTC). We still observe that only a small percentage of requests are impacted by timeouts and increased latency. At this stage, we have identified the issue as being related to network traffic between AWS and T Cloud Public. The root cause appears to be peering-related errors. Customers using our AWS-based offerings in combination with T Cloud Public-only products (e.g. d.velop sign) may experience intermittent issues. We will continue to provide updates as more information becomes available and will share the next update within 4 hours at the latest, or earlier if significant new insights arise.
(29. Jun, 09:51) Disrupted - We are currently observing increased latency and occasional timeouts affecting all products hosted on T‑Cloud Public (OTC). The issue started on 27 June at 09:30 UTC. A small percentage of requests are impacted by timeouts or delays. Due to resilient software design, the overall impact is low and most customers are unlikely to notice any disruption, although some may experience intermittent end‑to‑end issues. We are actively investigating the issue together with T‑Cloud Public (OTC) and will share further updates as soon as more information becomes available. We expect to provide the next update within 4 hours or earlier if significant new insights arise.