My Drupal 7 site has been up and running for a couple years without issue. I went to log in as admin today to check for security updates and had the following experience:
- When I first browse to the page (as anonymous), the site works fine.
- Enter login information to log in as admin, immediately greeted with "Access denied".
- Browse other pages in the same browser session, the entire site says "Access denied".
- Visit the website in a new browser session, a private browser session, a private browser session on a different computer, or a private browser session on a different computer at a remote site with a different IP address, all using 4 different browsers - all say "Access denied".
The first thing the internet said was to edit settings.php and modify the $cookie_domain setting, which I had left at default (unconfigured). I tried setting this to my domain but it made no difference.
After extensive troubleshooting, I have identified the following additional behavior:
- When I log in and get Access Denied, it only affects a particular subdomain even though it's the same site (example.com vs www.example.com).
- When the site is in a working state, the accesslog table shows anonymous browsing with uid of 0. After I attempt to log in, the accesslog shows activity with a uid of NULL. This makes some sense if there's no session in the sessions table to match the cookie in my browser.
- The watchdog table shows no errors, only "user Session opened for %name." and "access denied user/1".
- Anonymous access to the site returns to normal after a CRON job runs. Since I installed Drush I can now run CRON manually which speeds up troubleshooting.
- If I attempt to log in using correct credentials, it breaks the site again and access is denied (as described) until the next CRON job.
- If I attempt to log in using incorrect credentials, I get "unrecognized username or password" and the site continues to work fine for anonymous browsing.
So I started digging around:
I found the Anonymous user's "status" in the users table was set to 0(blocked). I set this to 1. No change.
I found that the row for the Anonymous user was completely missingfrom the users_roles table. I added this row with values 0,1 for uidand rid respectively. No change. (This row is also missing from a new install so it may not be necessary - I'm not an expert here.)
I verified that the role, role_permission, and node_access tableswere properly populated. No change.
I attempted to browse to /admin/content/node-settings/rebuild but get access denied on that, too.
I checked the accesslog table and recent entries from today show numerous logins with a uid of NULL.
I truncated the sessions table and when I attempt to browse or log in to the site no new rows are added to the table (similar to this question).
I installed drush and cleared the cache and ran the cron logout as suggested by the answer in the StackOverflow question. This reduced the problem from everyone getting access denied to only me getting access denied, resolved by clearing the site's session cookie. So it's an improvement in that attempting to log in doesn't break the site for everyone now. Still no rows in the sessions table.
I hashed a new password and set it directly in the database. No change in behavior.
I have attempted to reset my password using the password reset links. I receive the password reset e-mail. When I open the link, I get a new accesslog entry with uid of 0 and when I click "login" I get an accesslog entry with uid of 1 (my admin user), and then when I attempt to reset my password or go anywhere else on the site, I get accesslog entries with uid of NULL. Sessions table is still empty. This works the same if I use the command
drush ulito generate a login.I found this question/answer and checked my themes. Unable to access the admin pages, I checked in the database and found my current theme was set to 0 (disabled). I set this to 1 (enabled) as well as its prerequisites. No change.
I found this question/answer and attempted a repair on my sessions table. I had to set it o use MyASM instead of InnoDB to run the repair. Repair succeeds with no change to behavior.
I found this question/answer and disabled all modules using the provided drush command. No change in behavior. Other suggestions in that answer either do not apply to my situation (hosted platform) or likewise do not work.
I attempted to install a new instance of Drupal 7 on the same server. The new instance has the exact same problem. This leads me to believe it's something systemic on the server. Based on the above StackOverflow answer I suspect my hosting provider may have forced an update from PHP 5.3 to 5.4 without notifying me. I have now tried several PHP versions available from my hosting provider (5.4, 5.5, 5.6, and 7.0) in addition to all of the steps outlined above.
I have streamlined my troubleshooting process to the following steps:
- Make a change
- Clear browser cookies
- Use Drush to clear cache and run CRON
- Refresh website and clear cookies again if a session cookies shows up
- Test login -> Access Denied
- Check Watchdog and Sessions tables in PHPMyAdmin
So far nothing seems to be making any difference at all. The majority of suggestions I've found on the internet all point to clearing cookies or setting the $cookie_domain variable which make no difference.
Can anyone help me regain admin access to my site and get this working?
More notes
- I have created a duplicate of my site at an alternate URL for invasive testing.
- I have attempted a new install of Drupal 7.59 using my hosting provider's CPanel. Same behavior.
- I have attempted a new instance of Drupal 7.59 downloading directly from Drupal.com and running install.php. Same behavior.
This HAS to be something on the server itself.
At this point I'm looking at building a copy of my site on a VM at home to try and fix while I look for another hosting provider.
I'd like to migrate to Drupal 8, but that's a huge project - finding replacements for themes and modules that aren't supported in Drupal 8 and migrating data and content. I'm not looking forward to that with admin access broken.
Modules list from Drush:
Package Name Type Status Version Administration Module Missing Message Fixer (module_missing_message_fixer) Module Enabled 7.x-1.7 Administration Rename Admin Paths (rename_admin_paths) Module Enabled 7.x-2.3 Advanced CSS/JS Aggregation AdvAgg Async Font Loader (advagg_font) Module Enabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Bundler (advagg_bundler) Module Enabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg CDN CSS (advagg_css_cdn) Module Not installed 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg CDN Javascript (advagg_js_cdn) Module Not installed 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Compress CSS (advagg_css_compress) Module Enabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Compress Javascript (advagg_js_compress) Module Enabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg CSS/JS Validator (advagg_validator) Module Not installed 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg External Compression (advagg_ext_compress) Module Disabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Modifier (advagg_mod) Module Enabled 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Relocate (advagg_relocate) Module Not installed 7.x-2.30 Advanced CSS/JS Aggregation AdvAgg Subresource Integrity (advagg_sri) Module Not installed 7.x-2.30 Advanced CSS/JS Aggregation Advanced CSS/JS Aggregation (advagg) Module Enabled 7.x-2.30 CCK CCK (cck) Module Enabled 7.x-3.0-alpha3 CCK Content Migrate (content_migrate) Module Not installed 7.x-3.0-alpha3 Chaos tool suite Bulk Export (bulk_export) Module Not installed 7.x-1.12 Chaos tool suite Chaos tools (ctools) Module Enabled 7.x-1.12 Chaos tool suite Chaos Tools (CTools) AJAX Example (ctools_ajax_sample) Module Not installed 7.x-1.12 Chaos tool suite Chaos Tools (CTools) Plugin Example (ctools_plugin_example) Module Not installed 7.x-1.12 Chaos tool suite Custom content panes (ctools_custom_content) Module Not installed 7.x-1.12 Chaos tool suite Custom rulesets (ctools_access_ruleset) Module Not installed 7.x-1.12 Chaos tool suite Page manager (page_manager) Module Not installed 7.x-1.12 Chaos tool suite Stylizer (stylizer) Module Not installed 7.x-1.12 Chaos tool suite Term Depth access (term_depth) Module Not installed 7.x-1.12 Chaos tool suite Views content panes (views_content) Module Not installed 7.x-1.12 Core Aggregator (aggregator) Module Not installed 7.59 Core Block (block) Module Enabled 7.59 Core Blog (blog) Module Not installed 7.59 Core Book (book) Module Not installed 7.59 Core Color (color) Module Enabled 7.59 Core Comment (comment) Module Not installed 7.59 Core Contact (contact) Module Disabled 7.59 Core Content translation (translation) Module Disabled 7.59 Core Contextual links (contextual) Module Disabled 7.59 Core Dashboard (dashboard) Module Disabled 7.59 Core Database logging (dblog) Module Enabled 7.59 Core Field (field) Module Enabled 7.59 Core Field SQL storage (field_sql_storage) Module Enabled 7.59 Core Field UI (field_ui) Module Enabled 7.59 Core File (file) Module Enabled 7.59 Core Filter (filter) Module Enabled 7.59 Core Forum (forum) Module Not installed 7.59 Core Help (help) Module Disabled 7.59 Core Image (image) Module Enabled 7.59 Core List (list) Module Enabled 7.59 Core Locale (locale) Module Enabled 7.59 Core Menu (menu) Module Enabled 7.59 Core Node (node) Module Enabled 7.59 Core Number (number) Module Enabled 7.59 Core OpenID (openid) Module Not installed 7.59 Core Options (options) Module Enabled 7.59 Core Overlay (overlay) Module Disabled 7.59 Core Path (path) Module Enabled 7.59 Core PHP filter (php) Module Disabled 7.59 Core Poll (poll) Module Disabled 7.59 Core RDF (rdf) Module Enabled 7.59 Core Search (search) Module Enabled 7.59 Core Shortcut (shortcut) Module Disabled 7.59 Core Statistics (statistics) Module Enabled 7.59 Core Syslog (syslog) Module Not installed 7.59 Core System (system) Module Enabled 7.59 Core Taxonomy (taxonomy) Module Enabled 7.59 Core Testing (simpletest) Module Not installed 7.59 Core Text (text) Module Enabled 7.59 Core Toolbar (toolbar) Module Enabled 7.59 Core Tracker (tracker) Module Not installed 7.59 Core Trigger (trigger) Module Not installed 7.59 Core Update manager (update) Module Enabled 7.59 Core User (user) Module Enabled 7.59 Development Metatag: Devel (metatag_devel) Module Not installed 7.x-1.22 Feeds Feeds (feeds) Module Not installed 7.x-2.0-alpha8 Feeds Feeds Admin UI (feeds_ui) Module Not installed 7.x-2.0-alpha8 Feeds Feeds Import (feeds_import) Module Not installed 7.x-2.0-alpha8 Feeds Feeds News (feeds_news) Module Not installed 7.x-2.0-alpha8 ImageAPI ImageAPI (imageapi) Module Disabled 7.x-1.x-dev ImageAPI ImageAPI GD2 (imageapi_gd) Module Disabled 7.x-1.x-dev ImageAPI ImageAPI ImageMagick (imageapi_imagemagick) Module Not installed 7.x-1.x-dev Menu TB Mega Menu (tb_megamenu) Module Enabled 7.x-1.0-rc2 Multilingual - Panels translation (i18n_panels) Module Not installed 7.x-3.9 Internationalization Other Chosen (chosen) Module Disabled 7.x-1.0-beta6 Other Facebook comments (facebook_comments) Module Enabled 7.x-1.0 Other Fast 404 (fast_404) Module Enabled 7.x-1.5 Other Font Awesome (fontawesome) Module Disabled 7.x-1.0-beta6 Other Job Scheduler (job_scheduler) Module Not installed 7.x-2.0-alpha3 Other Job Scheduler Trigger (job_scheduler_trigger) Module Not installed 7.x-2.0-alpha3 Other Libraries (libraries) Module Enabled 7.x-2.3 Other Pathauto (pathauto) Module Enabled 7.x-1.3 Other Quicktabs (quicktabs) Module Enabled 7.x-3.8 Other Quicktabs Styles (quicktabs_tabstyles) Module Enabled 7.x-3.8 Other Site map (site_map) Module Not installed 7.x-1.3 Other Special menu items (special_menu_items) Module Enabled 7.x-1.0-beta6 Other Token (token) Module Enabled 7.x-1.7 Panels Mini panels (panels_mini) Module Enabled 7.x-3.9 Panels Panel nodes (panels_node) Module Enabled 7.x-3.9 Panels Panels (panels) Module Enabled 7.x-3.9 Panels Panels In-Place Editor (panels_ipe) Module Enabled 7.x-3.9 Performance and scalability APC - Alternative PHP Cache (apc) Module Disabled 7.x-1.0-beta6 Performance and scalability HTTP Parallel Request Library (httprl) Module Enabled 7.x-1.14 Radioactivity Radioactivity (radioactivity) Module Enabled 7.x-2.10 Radioactivity Radioactivity defaults (radioactivitydefaults) Module Not installed 7.x-2.10 Responsive Respond.js (respondjs) Module Enabled 7.x-1.5 Search Custom Search (custom_search) Module Enabled 7.x-1.20 Search Custom Search Blocks (custom_search_blocks) Module Not installed 7.x-1.20 Search Custom Search Internationalization (custom_search_i18n) Module Not installed 7.x-1.20 Search Custom Search Taxonomy (custom_search_taxonomy) Module Not installed 7.x-1.20 SEO Metatag (metatag) Module Enabled 7.x-1.22 SEO Metatag Importer (metatag_importer) Module Not installed 7.x-1.22 SEO Metatag: App Links (metatag_app_links) Module Not installed 7.x-1.22 SEO Metatag: Context (metatag_context) Module Not installed 7.x-1.22 SEO Metatag: Dublin Core (metatag_dc) Module Not installed 7.x-1.22 SEO Metatag: Dublin Core Advanced (metatag_dc_advanced) Module Not installed 7.x-1.22 SEO Metatag: Facebook (metatag_facebook) Module Enabled 7.x-1.22 SEO Metatag: favicons (metatag_favicons) Module Not installed 7.x-1.22 SEO Metatag: Google Custom Search Engine (CSE) Module Not installed 7.x-1.22 (metatag_google_cse) SEO Metatag: Google+ (metatag_google_plus) Module Not installed 7.x-1.22 SEO Metatag: hreflang (metatag_hreflang) Module Not installed 7.x-1.22 SEO Metatag: Mobile & UI Adjustments (metatag_mobile) Module Enabled 7.x-1.22 SEO Metatag: OpenGraph (metatag_opengraph) Module Enabled 7.x-1.22 SEO Metatag: OpenGraph Products (metatag_opengraph_products) Module Not installed 7.x-1.22 SEO Metatag: Panels (metatag_panels) Module Not installed 7.x-1.22 SEO Metatag: Twitter Cards (metatag_twitter_cards) Module Enabled 7.x-1.22 SEO Metatag: Verification (metatag_verification) Module Enabled 7.x-1.22 SEO Metatag: Views (metatag_views) Module Enabled 7.x-1.22 Sharing ShareThis (sharethis) Module Enabled 7.x-2.13 User interface CKEditor (ckeditor) Module Enabled 7.x-1.18 User interface Gallery Formatter (galleryformatter) Module Enabled 7.x-1.4 User interface jQuery plugins (jquery_plugin) Module Enabled 7.x-1.0 User interface jQuery Update (jquery_update) Module Enabled 7.x-2.7 User interface Superfish (superfish) Module Enabled 7.x-1.9 User interface Wysiwyg (wysiwyg) Module Enabled 7.x-2.4 Views Views (views) Module Enabled 7.x-3.18 Views Views Content Cache (views_content_cache) Module Enabled 7.x-3.0-alpha3 Views Views Slideshow (views_slideshow) Module Enabled 7.x-3.9 Views Views Slideshow Simple Pager (views_slideshow_simple_pager) Module Not installed 7.x-3.9 Views Views Slideshow: Cycle (views_slideshow_cycle) Module Enabled 7.x-3.9 Views Views UI (views_ui) Module Enabled 7.x-3.18 XML sitemap XML sitemap (xmlsitemap) Module Enabled 7.x-2.3 XML sitemap XML sitemap custom (xmlsitemap_custom) Module Enabled 7.x-2.3 XML sitemap XML sitemap engines (xmlsitemap_engines) Module Enabled 7.x-2.3 XML sitemap XML sitemap internationalization (xmlsitemap_i18n) Module Not installed 7.x-2.3 XML sitemap XML sitemap menu (xmlsitemap_menu) Module Enabled 7.x-2.3 XML sitemap XML sitemap node (xmlsitemap_node) Module Enabled 7.x-2.3 XML sitemap XML sitemap taxonomy (xmlsitemap_taxonomy) Module Enabled 7.x-2.3 XML sitemap XML sitemap user (xmlsitemap_user) Module Disabled 7.x-2.3 Core Bartik (bartik) Theme Disabled 7.59 Core Garland (garland) Theme Disabled 7.59 Core Seven (seven) Theme Enabled 7.59 Core Stark (stark) Theme Disabled 7.59 Other nucleus (nucleus) Theme Enabled 7.x-1.5 Other nucleus_starter (nucleus_starter) Theme Enabled 7.x-1.5 Other Tb Sirate (tb_sirate) Theme Enabled 7.x-1.1