Intermittent sign in interruption

Incident Report for Connectbase

Resolved

This incident has been resolved.

RCA:

Date:
September 30 – October 10, 2025

Impact
Intermittent slow performance and login issues in the Connectbase Portal and APIs
Summary
Between September 30 and October 10, some Connectbase customers experienced intermittent slowness and login errors when using the Connectbase platform. During peak impact periods, users saw delays when logging in, switching companies, or generating quotes. All services were restored to normal performance by October 11. The issue was traced to a combination of background database activity and heavy user workloads that temporarily exceeded our database's ability to manage concurrent updates.

Root Cause
A background data synchronization task—used to keep our location and network data aligned across systems—generated a continuous stream of write activity in our database. This process coincided with other high-volume operations such as large location imports and pricing updates. While each process was functioning as designed, their combined load created contention inside the database, leading to delayed responses for user actions such as login, quote creation, and data updates. When the background replication process was paused, performance returned to normal immediately. Subsequent testing confirmed that the replication load was the primary trigger, amplified by concurrent import and pricing workloads.

Resolution
The replication service was paused on October 9, immediately restoring platform performance.
A configuration change was deployed to optimize large imports during high-traffic hours.
Database configuration settings were further tuned to improve write handling and memory efficiency.
The replication task was restarted in a controlled manner with controlled batch sizes and monitoring.
No customer data was lost or corrupted.
The system has remained stable since October 10 with normal performance across all environments.

Preventive Actions
To ensure this issue does not recur, Connectbase has implemented and planned the following actions:

Reduced replication batch size and adjusted database flush parameters - Completed
Added import scheduling guardrails to defer large data uploads during business hours - Completed
Partitioning large database tables to improve write performance - In Progress
Isolating background replication and import workloads to a dedicated database cluster - Planned (Q1 2026)
Expanding monitoring to include per-process resource usage and lock analysis - In Progress

Customer Impact
Duration: Intermittent from September 30 – Oct 10
Services affected: Connectbase Web Portal (The Connected World), APIs, Authentication, and Quoting workflows
Resolution time: Intermittent system performance degradation over this period with full service restoration within minutes of replication suspension
Data integrity: No loss or corruption occurred

Next Steps
Connectbase is taking a broader architectural approach to improve resilience, including separating operational, analytical, and synchronization workloads. We will continue to monitor replication and import processes closely and will communicate any related maintenance windows proactively.
Posted Oct 10, 2025 - 17:14 EDT

Monitoring

System performance is stable. We are actively investigating the recent performance issues and continue to review and analyze system behavior.
Posted Oct 09, 2025 - 11:00 EDT

Update

We are currently investigating an issue causing intermittent login interruptions and slowness across the platform. Some users may be experiencing delays when accessing or navigating the system.
Our engineering team is actively working to identify the root cause and restore full performance. We will continue to share updates as we make progress.
We apologize for the inconvenience and appreciate your patience and understanding as we work to resolve this issue.
Posted Oct 09, 2025 - 08:46 EDT

Update

System performance is stable. We are actively investigating the recent performance issues and continue to review and analyze system behavior.
Posted Oct 06, 2025 - 08:28 EDT

Update

We are actively investigating the ongoing performance issues and continue to review and analyze system behavior. We sincerely apologize for the inconvenience and appreciate your patience and understanding as we work toward a resolution.
Posted Oct 03, 2025 - 16:43 EDT

Update

We are actively investigating this issue and will continue to provide updates as progress is made. We sincerely apologize for the inconvenience and appreciate your patience and understanding as we work toward a resolution
Posted Oct 03, 2025 - 15:08 EDT

Update

We are actively investigating this issue and will continue to provide updates as progress is made. We sincerely apologize for the inconvenience and appreciate your patience and understanding as we work toward a resolution.
Posted Oct 03, 2025 - 12:42 EDT

Update

We are continuing to investigate this issue.
Posted Oct 03, 2025 - 10:37 EDT

Update

We are currently investigating an issue causing intermittent login interruptions and slowness across the platform. Some users may be experiencing delays when accessing or navigating the system.
Our engineering team is actively working to identify the root cause and restore full performance. We will continue to share updates as we make progress.
We apologize for the inconvenience and appreciate your patience and understanding as we work to resolve this issue.
Posted Oct 03, 2025 - 08:52 EDT

Update

Dear Connectbase Customer,

We are continuing to work on resolving the performance issues affecting login and other services. Our engineering team is actively implementing remediation steps to restore full functionality and improve platform stability.

Thank you for your patience. We'll keep you updated as we make progress.
Posted Oct 02, 2025 - 14:13 EDT

Update

We’re aware of an issue preventing some users from logging in or accessing the platform. Our team is actively working to resolve the issue. We’ll provide updates as soon as we have more information. We appreciate your patience.
Posted Oct 02, 2025 - 08:58 EDT

Investigating

We’re aware of an issue preventing some users from logging in or accessing the platform. Our team is actively working to resolve the issue. We’ll provide updates as soon as we have more information. We appreciate your patience.
Posted Oct 02, 2025 - 08:57 EDT

Monitoring

The platform is currently stable and we're actively monitoring.
Posted Sep 30, 2025 - 17:52 EDT

Investigating

Users may experience intermittent login delays, time-outs, or slower than usual performance across the platform. Our team is actively working on the issue. We’ll provide updates as soon as we have more information. We appreciate your patience.
Posted Sep 30, 2025 - 17:02 EDT
This incident affected: The Connected World.