EE compatibility updates:

  • Publisher is EE5 compatible, but it does not currently support the Fluid field.

ExpressionEngine.com licenses:

  • If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: 500 Errors and static caches not appearing to be fully cleared until viewed

Status Client Wait
Add-on / Version Speedy 1.1.1
Severity
EE Version 5.3.2

Mighty Citizen

Dec 10, 2020

I’m working with a client whose heavily trafficked government site is experiencing some bugs that we cannot replicate on our development sites. I’m wondering if / hoping you may be amenable to participating in a Zoom to quickly see some of the the issues that are happening on their production site in real time. We’re hoping you may recognize some red flags that would either help us figure out the problem or at least get some ideas from you on how best to replicate the problems in a dev environment so that our troubleshooting could be more helpful.

Some of the issues they’ve reported with Speedy include:

- Multiple front-end pages display content of other pages, as if static cache files are not deleting properly or are not overwriting old static files properly.

- Speedy is only clearing the expired items when the client views the static items in the Speedy CP UI. More clearly, when the client attempts to use the Speedy clear cache URL, it fails. They get the error message: “Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in ...\system\ee\EllisLab\ExpressionEngine\Library\Data\Entity.php on line 654” (we’ve bumped up the PHP memory limit to 512MB in an effort to avoid this, but that did not help). What’s odd is that prior & immediately following the error, the Speedy Control Panel read the same number of items (a large number, ~1,000-2,000). But then when you view the static items the number drops drastically (~100-200).

- Speedy is only intermittently updating static pages on an entry’s save. It seems to work on the site’s Generic Page entries but not in other Channels (News & Home Page). Another Channel (Organization Pages) seems to update the state page cache intermittently.

- 500 errors are occurring on the first entry save of the day, as if Speedy is clearing out all the expired files at the same time. They have set up a cron job to clear out the static folder every morning at 5am, but our client logs in each morning to initiate the first Save of the day to prevent others from possibly getting a 500 error. But as the issue has continued, it’s currently taking multiple attempts save an entry without a 500 error even after viewing the static files in the Speedy CP.

- Another error they’ve seen in the Speedy CP: “Exception Caught SplFileInfo::getMTime(): stat failed for .../static/default_site/static/organization/disabilities/disability-proclamations/index.php . . . user/addons/speedy/Service/Drivers/AbstractFilesystemDriver.php:59”

It seems as if Speedy continues to register cached files until you view the cache in the CP. But that’s a separate problem from the 500 errors.

I will need to get *your* IP Address in order to whitelist you for access to the dev site and dev server.

Thank you!

Wiley Koepp

#1

BoldMinded (Brian)

Dec 10, 2020

This is a lot to take in. My IP address will be in the next comment, but I’d prefer to work through this without me having to login to anything. Coming into someone else’s site cold it time consuming and requires a lot of mental gynmastics. I also can’t step debug on remote servers, var_dumping and saving files through FTP is also very time consuming.

Can you attach some screenshots of your Speedy config and cache busting rules? This kind of sounds like a disk I/O issue. For some reason Speedy is trying to delete a lot of files at once, or re-warm a lot of pages at once. Seeing the cache busting rules will help. If it’s trying to clear out a lot of files at a single time, perhaps the cache busting rules are too vague. You could var_dump in the CacheBreaker.php->_breakEntryCache() method to see how many items its trying to clear at a single time.

You say there are 1-2000 cache items? So there are that many files in the static folder? Have you considered using Redis as the static cache option? It is much more performant and you won’t have any disk I/O errors.

#2

BoldMinded (Brian)

Dec 10, 2020

Comment has been marked private.

#3

BoldMinded (Brian)

Dec 10, 2020

Comment has been marked private.

#4

Mighty Citizen

Dec 11, 2020

Comment has been marked private.

#5

Mighty Citizen

Dec 11, 2020

Comment has been marked private.

#6

BoldMinded (Brian)

Dec 11, 2020

I assume the IP address has not been added b/c I can’t access the site or FTP?

#7

Mighty Citizen

Dec 11, 2020

Comment has been marked private.

#8

Mighty Citizen

Dec 11, 2020

Comment has been marked private.

#9

BoldMinded (Brian)

Dec 11, 2020

Comment has been marked private.

#10

BoldMinded (Brian)

Dec 11, 2020

BTW, I can get to the admin login page, but the credentials provided don’t work. I also can’t login to the FTP server either.

#11

Mighty Citizen

Dec 14, 2020

Comment has been marked private.

#12

BoldMinded (Brian)

Dec 14, 2020

I just realized its the _breakCache function that is called with the ACT url, so this is what should be changed in the CacheBreaker.php file around line 90.

public function _breakCache()
    {
        // This is a potentially long running operation, disable the time limit so PHP won't timeout.
        set_time_limit(0);

        $clear_tags = [];
        $clear_items = [];
        $refresh_items = [];

        $can_refresh = $this->canRefreshItems();
        error_log('_breakCache $can_refresh ' . boolval($can_refresh));

        // Collect all tags and tagged items.
        foreach ($this->collectAllTags() as $tag) {
            $clear_tags[] = $tag->tag;
            $clear_items[] = $tag->key;
            if ($can_refresh) {
                $refresh_items[] = $tag->key;
            }
        }

        // Collect all items from all drivers.
        foreach ($this->collectItemsFromPath('') as $item) {
            $clear_items[] = $item;
            if ($can_refresh) {
                $refresh_items[] = $item;
            }
        }

        $clear_tags = array_unique($clear_tags);
        $clear_items = array_unique($clear_items);
        $refresh_items = array_unique($refresh_items);

        try {
            if (count($clear_tags)) {
                error_log('_breakCache clear_tags:');
                error_log(json_encode($clear_tags));
                $this->clearTags($clear_tags);
            }

            if (count($clear_items)) {
                error_log('_breakCache claer_items:');
                error_log(json_encode($clear_items));
                $this->clearItems($clear_items);
            }

            if (count($refresh_items)) {
                error_log('_breakCache refresh_items:');
                error_log(json_encode($refresh_items));
                $this->refreshItems($refresh_items);
            }
        } catch (\Exception $exception) {
            error_log('_breakCache exception:');
            error_log($exception->getMessage());
        }
    }
#13

BoldMinded (Brian)

Dec 14, 2020

Comment has been marked private.

#14

BoldMinded (Brian)

Dec 14, 2020

Based on that file path, it looks like this is a Windows server, and the CP is showing 7128 static cache items, but listing none when attempting to view them. Are there really 7000 pages on this site? Seems like it’s reading from the wrong directory? I know this might take a bit of work, but do you have an Linux server you could put the site on just to test and see if things work there? If so then I can focus on it possibly being a Windows server or file system issue. I’d like to fix this, but if you can switch to Redis it’ll be better for you all around. If you have more than 1 web server than static file caching is not very optimal as it creates a cache file on each server, and it won’t clear cache files on multiple servers, just the server you make the request from. Using a separate Redis instance is more scalable.

#15

Mighty Citizen

Dec 22, 2020

Comment has been marked private.

#16

Mighty Citizen

Dec 22, 2020

Comment has been marked private.

#17

BoldMinded (Brian)

Dec 22, 2020

The linux site doesn’t help much if the issue only happens on the Windows box. I still can’t FTP to it using the credentials provided. Does it use a non-typical port number? Transmit just doesn’t connect.

#18

Mighty Citizen

Dec 22, 2020

Comment has been marked private.

#19

BoldMinded (Brian)

Dec 22, 2020

Yeah, that is definitely what I tried, still didn’t work.

Login to reply