Sorry I am using Redis

I am using fragment caching

Yes, it is only an issue in the control panel. I am not sure how large the count is, and it may be difficult for me to var_dump on a live site, but I will run that query in the database and determine how many rows it returns.

I added another image, but it just returns 1 for me. So that doesn’t seem to be the issue. Although it is some count statement somewhere causing this from the first image.

Asking our senior developer….

Comment has been marked private.

Okay, so I am pretty certain I found the offending script/s in Redis Driver:

// line 96
public function countItems()
{
  return count($this->getItemsFromPath('/'));
}

//which gets called by countItems() on line 208
public function getItemsFromPath($path)
{
   $path = $this->prefix . '/' . ltrim($path, '/');
   $path = rtrim($path, '/') . '/';

   $pattern = sprintf('~^%s~', preg_quote($path, '~'));
   $iterator = $this->getIterator($pattern);

   $items = [];
   $path_len = strlen($path);

   // Strip the $path prefix from each filename
   foreach ($iterator as $key => $data) {
       $items[] = substr($key, $path_len);
   }

   sort($items, SORT_STRING);

   return $items;
}

// which gets called by getItemsFromPath() on line 235
private function getIterator($pattern = null)
    {
        if ($pattern === null) {
            $pattern = sprintf('~^%s~', preg_quote($this->prefix, '~'));
        }

        $iterator = null;
        $store = [];

        try {
            while($keys = $this->client->scan($iterator, '*')) {
                foreach($keys as $key) {
                    if (preg_match($pattern, $key)) {
                        $store[$key] = $this->client->hGetAll($key);
                    }
                }
            }
        } catch (RedisException $exception) {
            // Fail silently...
        }

        return new \ArrayIterator($store);
    }

This series of functions grabs all items with ‘*’ then it has a nested for loop that checks path pattern, then it loops through and strips path, then it sorts, and then it counts. I am not great at calculating runtime analyses so I won’t attempt, but I would guess that very much is not a scalable function. This is for sure what is causing the issue. Here is a much better way to count redis using the php redis library: https://www.php.net/manual/en/function.substr.php.

<pre><code> public function countItems() { return $this->client->dbSize(); } [\code]

There are still a lot of instances where this spider web of nested loops is getting called. Such as deleteItem() or clear(). These should all be rewritten. I pushed my above change to my server to help out, but will watch for more bugs.

From the first sentence above *so you should have access now. I just tested that you did”

Both have the same settings for max_execution_time=30, memory_limit=500M, max_input_time=30

Are there possible problems with speedy if I directly flush db? I have a pipeline that flushes redis database and does not go through the speedy addon.

So, another update on this issue that might be revealing.

We were having issues with saving an entry in the open status due to the size of the exp_speedy_tags table. Basically, the entry would timeout when trying to save. However, after looking at the exp_speedy_tags table had grown to 125,000 entries even thought my cache was below 2000 items. After deleting the entries in this table, saving entries was lightening fast. This would explain why when I added a large amount of cache items on my staging environment, it was working correctly.

This is specifically if a cache breaking rule is set on a channel with a large amount of entries in the exp_speedy_tags, and it does not have an effect on overall clearing of the cache nor actual users loading urls on the front-end. The issue here seems to be that if a site has many urls and a decent amount of traffic, the amount of tags will eventually become to large for cache breaking.

So, the exp_speedy_tags table obviously serves the purpose of keeping track of tags for clearing on the add-on overview panel and also for cache breaking rules, and possibly for identifying if a cache item already exists without contacting the redis server, but it doesn’t seem obvious that it is really is necessary in the end. Why not just add segments to the end of a cached item like so,

speedy/default_site/local/news/article/a-human-future/article-a-human-future/tags/news/news-article

Then all searches for tags could simply grab the items by the end segments with a regex check. Also, a redis command to check for that key item could be done very simply on front end page loads. Sometimes, this may mean that cache items are decently long, but I don’t see why that would be a performance issue. This would solve the issue that I am facing since the cache would never get large enough to cause issues.

Will do some more investigation to see if this would be possible, and whether there are other reasons for not doing this, but throwing it up here for the time being.

Also as a side note, I changed Redis Clear function from

public function clear()
    {
        $keys = $this->getItemsFromPath('/');

        foreach($keys as $key) {
            $this->client->del($this->getKeyPath($key));
        }

        return true;
    }

To the following:

public function clear()
    {
        $this->client->flushDb();

        return true;
    }

Besides that and the countItems function. There are no other hacks.

Cool will be looking into clearing table as well. Thanks for the help.

I was not suggesting you change the code to flushDB() was just pointing out that on my environment that is how I have it set to avoid the getItemsFromPath function. If I am only using my redis server for speedy, this shouldn’t be a problem for us?

Cool, I am good to close this one. I think you were right about the pipeline causing issues going to try to use the action call for breaking all cache instead of a redis flushall. Also, another issue seemed to be that our client and their team has been clearing the redis driver specifically and not the clear all button which explains why the exp_speedy_tags table kept getting so big. Going to restrict access to avoid this issue.

Anyways, thanks so much for your help. I really appreciate it, and have a good weekend.

Thanks Patmos