12.04 Troubleshooting Common Issues

Why this is a reference article

This article is a runbook. When something breaks, find the symptom in the table of contents below, jump to the section, work through the diagnostic steps. The patterns recur; once you've debugged each one once, the second time is fast.

Issue 1: Install fails with "Database connection error"

Symptom: the installer can't connect to the database; the install can't proceed.

Likely causes (in order of frequency):

1. Wrong credentials in simplerisk/includes/config.php. Verify by trying the credentials manually: mysql -u -p -h .
2. Database server unreachable. Check network: nc -zv 3306 should succeed.
3. Database user doesn't have access from the SimpleRisk server's IP. Check SHOW GRANTS FOR ' '@' '; .
4. Database doesn't exist. The configured simplerisk database needs to exist. Create with CREATE DATABASE simplerisk;.
5. MySQL version too old. SimpleRisk requires MySQL 5.7+. Check with mysql --version. (MariaDB is no longer supported — see System Requirements.)

Issue 2: User can't log in (correct credentials)

Symptom: user enters correct username and password, login fails.

Likely causes:

1. Account is locked (failed-attempts threshold reached). Check SELECT lockout FROM user WHERE username = ' '; . If 1, unlock via User Management or UPDATE user SET lockout = 0 WHERE username = ' '; .
2. Password expired (password policy enforces expiration). Force a password change via the admin reset path.
3. MFA misconfigured (user enrolled but lost device). See Multi-Factor Authentication recovery procedure.
4. SSO misconfigured (the user is type='ldap' or type='saml' and the SSO path is broken). Test SSO directly; check the Authentication Extra's settings.
5. Session storage is broken (database can't write sessions rows). Check database disk space and table integrity.

Issue 3: Cron jobs aren't firing

Symptom: workflows don't fire, notifications don't send, AI jobs queue indefinitely.

Diagnostic steps:

1. Check cron daemon is running: systemctl status cron (or your distribution's equivalent).
2. Check the SimpleRisk cron entries: crontab -l (under the user that runs SimpleRisk's cron).
3. Check cron_history table: most recent entries should be within the expected interval.
4. Check the debug log: cron job runs log entries; failures log errors. Filter for the specific job's output.
5. Run the cron job manually: php /path/to/simplerisk/cron/cron_notification.php (or the relevant script). Observe the output for errors.
6. Check filesystem permissions: the cron user needs read/write access to simplerisk/.

For the cron queue worker specifically: check that cron_queue_worker.php is running. On some installs it runs continuously; on others it's invoked by cron.

Issue 4: Slow page loads

Symptom: pages take > 5 seconds to load.

Diagnostic steps:

1. Identify which pages are slow: all pages, or specific ones?
2. Check PHP opcache: php -i | grep opcache.enable should show On. See Performance Tuning.
3. Check PHP-FPM worker status: pm.max_children saturation produces queueing.
4. Check MySQL slow query log: enable, capture queries from a slow page load, identify the culprit.
5. Check server resources: top, free -h, df -h. Saturated CPU, OOM, or full disks all produce slowness.
6. Check database server resources if separated.

For specific slow operations, see Performance Tuning.

Issue 5: Notifications not arriving

Symptom: SimpleRisk should send an email, the email doesn't arrive.

Diagnostic steps:

1. Check SMTP configuration: Configure → Settings → Mail Settings. Send a test email; observe.
2. Check the test email succeeded: if not, the SMTP configuration is wrong.
3. Check the Notification Extra is active: Configure → Extras → Notification Extra.
4. Check the notification cron is running: cron_notification.php writes to the debug log on each run.
5. Check the notification queue: notification_sent_log table for queued-but-not-sent entries.
6. Check the recipient's spam folder: deliverability issues sometimes look like non-delivery.
7. Check the SMTP service's delivery logs: bounces, suppressions, rate limits.

For SMTP-side issues, see Email and the Notification Extra.

Issue 6: Encrypted data displays as ciphertext

Symptom: risk subjects, descriptions, etc. display as long base64-looking strings instead of readable text.

Likely causes (when the Encryption Extra is active):

1. Master key file missing or wrong on the server handling the request. Check simplerisk/extras/encryption/includes/init.php exists and contains the expected key.
2. Multi-server deployment with key file inconsistency. One server has the right key; others don't. Check all servers.
3. Master key file corrupted. Restore from your backup of the key file.

Without the master key, the data can't be decrypted. See Key Management and Rotation.

Issue 7: Workflow / AI jobs queued but not processing

Symptom: workflows fire (workflow_executions row created) but never complete; AI jobs queue but don't return results.

Diagnostic steps:

1. Check the cron queue worker is running: cron_queue_worker.php should be invoked regularly (or running continuously, depending on the install).
2. Check workflow_executions.status: stuck pending indicates the worker isn't picking up jobs; failed indicates execution errors.
3. Check the debug log: worker errors log there.
4. Check the worker's process: ps aux | grep cron_queue_worker should show it running (if continuous mode).
5. For AI specifically: verify the AI provider's API is reachable and the API key is valid (test via curl).

Issue 8: Database disk filling up

Symptom: monitoring alerts on database disk approaching full.

Diagnostic steps:

1. Identify the largest tables: SELECT table_name, ROUND((data_length + index_length) / 1024 / 1024) AS size_mb FROM information_schema.tables WHERE table_schema = 'simplerisk' ORDER BY size_mb DESC LIMIT 10;.
2. Common offenders: audit_log, debug_log, sessions (if not cleaned up), notification_sent_log.
3. For audit/debug logs: define and apply retention policy.
4. For sessions: ensure session GC is running; manually purge old: DELETE FROM sessions WHERE access < UNIX_TIMESTAMP() - 86400;.
5. Coordinate with operations: increase storage, archive old data, or both.

Issue 9: Integration receives 401 / 403

Symptom: API integration that worked yesterday returns 401 or 403 today.

Likely causes:

1. API key revoked or rotated without updating the integration.
2. User account disabled (the user the key belongs to).
3. API toggle disabled: Configure → Settings → API → check Enabled.
4. Permission revoked for the user (returns 403).
5. Hitting a path the user can't see (team filtering returns 403 or 404).

Check the SimpleRisk debug log for authentication failure entries; check the user's status; verify the key against current api_keys.

Issue 10: Upgrade fails partway

Symptom: SimpleRisk upgrade flow fails mid-process; install left in inconsistent state.

Diagnostic steps:

1. Read the upgrade error message carefully. Most upgrade failures are specific (database constraint, missing PHP extension, schema change conflict).
2. Check the upgrade function in simplerisk/includes/upgrade.php for the version that failed; trace the specific operation.
3. Restore from backup if data integrity is at risk. See Database Backup and Restore.
4. Re-run the upgrade after fixing the cause; SimpleRisk's upgrade flow is idempotent in most cases.
5. Contact support if the upgrade can't be made to complete; see When to Contact Support.

Issue 11: SAML / LDAP authentication broken

Symptom: SSO users can't log in; local users still can.

Diagnostic steps:

1. Check the Authentication Extra is active: Configure → Extras → Authentication Extra.
2. Test SAML / LDAP configuration: the Extra exposes test buttons; use them.
3. Check the IdP side: SAML metadata expired? LDAP service account credentials rotated? Network connectivity to the IdP?
4. For SAML: certificate expirations are common. Check SAML_METADATA_URL returns valid metadata.
5. For LDAP: test_ldap_configuration() (via the Extra's UI) reports the specific failure.
6. Check the debug log for SSO-related error entries.

If users are locked out entirely (SSO broken, no local accounts), the recovery path is to log in as the original local admin (whose credentials predate SSO activation).

Issue 12: Custom fields disappeared after upgrade

Symptom: after upgrade, custom fields aren't appearing on forms.

Diagnostic steps:

1. Check the Customization Extra is still active: upgrades occasionally deactivate Extras; reactivate.
2. Check custom_fields table: the field definitions should be present.
3. Check the form layout template in custom_template: layout may need to be re-saved.
4. Reload the page: browser cache may be serving an old form.

Custom fields rarely disappear on upgrade; if they do, restore the Customization Extra's data from backup.

Generic diagnostic checklist

For any issue not covered above:

1. What changed recently? Software upgrade, configuration change, infrastructure change, traffic shift.
2. What does the debug log say? Filter by time range around when the issue started.
3. What does the audit log say? Specific to entity-state issues.
4. What does the web server log say? PHP errors, 5xx responses.
5. What does the database log say? Slow queries, connection errors, errors.
6. Can you reproduce in non-production? If yes, debug there.
7. Does the issue affect all users or specific ones? If specific, what's different about them?
8. Does the issue happen at specific times? If yes, correlate with cron schedules, backup windows, traffic patterns.

Common pitfalls

A handful of patterns recur with troubleshooting.

• Restarting before debugging. Restart often hides the cause; debug first, restart after.

• Trying multiple fixes simultaneously. When something works, you don't know which fix did it. Fix one thing at a time.

• Trusting user reports over evidence. "It's broken" needs a reproducer. Get specifics before debugging.

• Skipping the logs. The logs usually contain the answer. Always check them first.

• Treating intermittent issues as fixed when they don't repeat for an hour. Intermittent issues recur. Document and continue investigating.

• Not coordinating with the program team. A "just trying to fix this" change in production during a workflow can disrupt users.

• Not having a rollback path. Before any production change, know how to undo it.

• Forgetting to monitor after the fix. A fix that "works" then breaks again two hours later wasn't actually a fix.

• Documenting fixes only in chat. Future operators (and you in 6 months) will need to find this. Document properly.

• Not asking for help when stuck. When to Contact Support.

Related

• Monitoring SimpleRisk
• Performance Tuning
• Scaling Considerations
• When to Contact Support
• The Debug Log
• The Audit Trail
• The Cron Jobs
• Database Backup and Restore