Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: Publisher Low Search

Status	Resolved
Add-on / Version	Publisher 1.0.3
Severity	Critical
EE Version	2.7.2

Peter Lewis

Jun 09, 2014

Apologies if you’d prefer this ticket via GitHub, but assumed this would be the better place to file an issue, and also appreciate that the extension is a freebie!

I’m using Publisher alongside Low Search, and Low Search now supports category keyword indexing, but wasn’t working. I’ve already gone through Low support to identify and track down this issue and alas tracked it down to your extension not supporting categories. Although looking at your commits, it states it was added in December (https://github.com/litzinger/publisher_low_search/commits/master). The problem comes when building the indexes - it simply doesn’t include any category information, whereas without the extension installed it does.

Any ideas how we can get this working?

Peter

Brian Litzinger

Peter, can you please provide a full example of your search form and results page, as well as an example of what your entry data looks like and how you’re searching on it? An actual link to a search page I can see would help. Also include what entries should appear in the results but are not b/c of the categories not working.

Peter Lewis

Hi Brian, there’s no point in showing the form or the results - it’s the creation of the collection that’s failing because of the extension hook preventing categories from being indexed.

Here’s the original post I created: http://expressionengine.stackexchange.com/questions/22505/low-search-and-category-keywords/23542#23542

Here’s the detailed post on Low Search Support (with the examples you require): https://getsatisfaction.com/low/topics/searches_including_category_keywords_not_working

These SQL commands seem to be the issue:

SELECT c.*, cp.entry_id
FROM (`exp_publisher_categories` c)
INNER JOIN `exp_category_posts` cp ON `c`.`cat_id` = `cp`.`cat_id`
WHERE `cp`.`entry_id` IN (0, 1, ... stripped due to max field length of 5000 characters for support site..., 224)

The first Low Search Index:

REPLACE INTO `exp_low_search_indexes` (`collection_id`, `entry_id`, `site_id`, `index_text`, `index_date`, `publisher_lang_id`, `publisher_status`) VALUES ('1', '396', '1', '| ferienparadies natterer see | ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| ferienparadies natterer see | ferienparadies natterer see |\n| au0060 | au0060 | au0060 |\n| 11 34201 47 23755 | 11 34201 47 23755 |\n| 43 051 254 6732 | 43 051 254 6732 |\n| 051 254 673216 | 051 254 673216 |\n| g01 | g01 |\n| natters | natters |\n| tirol | tirol |\n| tirol | tirol |\n| austria | austria |', 1402388592, '1', 'open')

Without the Publisher - Low Search Support Extension, it looks like this:

SELECT c.cat_id, c.group_id, c.cat_name, c.cat_description, fd.*, cp.entry_id
FROM (`exp_categories` c)
LEFT JOIN `exp_category_field_data` fd ON `c`.`cat_id` = `fd`.`cat_id`
INNER JOIN `exp_category_posts` cp ON `c`.`cat_id` = `cp`.`cat_id`
WHERE `cp`.`entry_id` IN (396, ... stripped due to max field length of 5000 characters for support site... , 495)

With the index looking like this:

REPLACE INTO `exp_low_search_indexes` (`collection_id`, `entry_id`, `site_id`, `index_text`, `index_date`) VALUES ('1', '396', '1', '| ferienparadies natterer see | ferienparadies natterer see | ferienparadies natterer see |\n|

 bar | bar |\n| disabled facilities | disabled facilities |\n| wifi all site | wifi all site |\n| wifi chargeable | wifi chargeable |\n| wifi partial site | wifi partial site |\n| pets | pets |\n| washing machine | washing machine |\n| club | club |\n| tv room | tv room |\n| shop | shop |\n| resturant | resturant |\n| hot food takeaway | hot food takeaway |\n| accomodation to rent | accomodation to rent |\n| lake | lake |\n| river beach | river beach |\n| kevin | kevin |\n| kids | kids |\n| dog friendly | dog friendly |\n| nature | nature |\n| walking | walking |\n| golf | golf |\n| golf nearby | golf nearby |\n| fishing | fishing |\n| bicycle hire | bicycle hire |\n| riding nearby | riding nearby |\n| lake | lake |\n| river swim | river swim |\n| childrens play area | childrens play area 

|\n| au0060 | au0060 | au0060 |\n| 11 34201 47 23755 | 11 34201 47 23755 |\n| 43 051 254 6732 | 43 051 254 6732 |\n| 051 254 673216 | 051 254 673216 |\n| g01 | g01 |\n| natters | natters |\n| tirol | tirol |\n| tirol | tirol |\n| austria | austria |', 1402388817)

I’ve broken the output so you can see all the new additions of the category data - these are only the category titles. No idea why the entry ID’s are different, I did the same index rebuild both times and it got the same entry for the first example?!?

Alas, can’t grant access to the build as the server is IP restricted. I’m assuming it can be replicated in any build using the latest Low Search and Publisher Extension…? I may be able to re-create on a completely different build, but let me know if there’s anything else I can provide.

Many thanks,

Peter

BoldMinded (Brian)

Thanks for the info. I’ll try to look into this more soon… it’ll probably be a few days.

Peter Lewis

No worries - thanks.

BoldMinded (Brian)

Peter, I figured out what the issue is and am close to having it fixed. Should have something in the next couple days, or early next week at the latest.

Peter Lewis

Brilliant - thank you for the update.

Brian Litzinger

Try downloading the new version from GitHub, I think its fixed now.

Peter Lewis

Afraid not.

Installed new version of the extension, v1.0.4 showing. Went to Low Search -> collections and rebuilt the index. No categories included in the exp_low_search_indexes table.

Set Publisher->Settings->Is Enabled? to No Disable the Publisher Low Search Extension Go to Low Search -> collections and reindex - categories now appearing in the Index.

BUT…

I enable Publisher but don’t enable the Publisher Low Search Extension. I rebuild the indexes and they are correctly built. I run a search and it correctly finds the category information. So why am I using the Publisher Low Search Extension? I can’t find any documentation as to why I had a problem in the first place and why I chose to install the Extension.

As you can tell - I’m confused now - What is the extension supposed to do?

Thanks,

Peter

BoldMinded (Brian)

The extension indexes the language data. It’ll work fine without the extension, but only for the default language content.

Not sure what else to say about the extension, I spent 3 hours working on fixing the query, and I added 4 category assignments to a couple of entries and all the custom field data from those categories started showing in the index for the entry. Did you double check that that categories are assigned to the entry for each language? Are the categories translated (did you go to the Publisher > Categories page)?

#10

Peter Lewis

Hi Brian,

OK, thanks for the clarification - I’d installed it on the build initially and just got really confused as to what it’s purpose is.

Yes categories are assigned to the entries - there are over 50 categories, each entry has around 10 categories and for my testing I added a unique description “kevin” to a category that is assigned to 211 entries. re-indexing in Low Search means that no category information is added to the low search index table.

I’ve just been testing in English, and although 3 other languages are setup, the categories aren’t translated in any of them - could that be throwing it off?

#11

BoldMinded (Brian)

It should still grab the categories as long as they are assigned to the entry. So edit a couple entries in other languages and make sure to save them with the cats assigned, then make sure those categories have a translation value for each custom field and it’s title, then try reindexing.

#12

BoldMinded (Brian)

What version of publisher are you using? Ticket says 1.0.3… I assume you’re on 1.4 and 1.0.3 refers to the extension version?

#13

Peter Lewis

I wrote out a huge essay response detailing all the findings as I’ve just spent the last 4 hours debugging and solving, alas the login had timed out, so when I submitted I lost the lot. Sigh.

So you’ll have to get the short version…

The problem is line 210

$entry_ids = array_keys($entries);

Array keys are just the default numerical array numbers - not entry IDs. It works on your build as I expect you’re testing with low level entry IDs, whereas my entries for the relevant channel start at 398.

So this needs to be replaced with:

$entry_ids = array();
foreach ($entries as $entryData)
    $entry_ids[] = $entryData["entry_id"];

Which then works. Sort of. Alas the excessive code looping means 227 entries with multiple categories (at least 10 per category) results in the foreach loop on line 310 being run 5,140,188 times. The previous response had a full description and breakdown of this calculation, which I won’t go into now. Oddly you’re pulling the draft data - which shouldn’t be included surely, so this would reduce the above loop to half, but I’m testing with only 2 languages, so if I had 4 it would double it again. Either way, the result is a memory limit fatal error and the only way to test was to reduce the query on line 304 with a limit to bypass the fatal error (the SQL used for the initial loop currently returns 5661 results, second foreach x227, then the languages x2, then the statuses x2).

There’s more optimisation required on lines 172 - 178 which seems to have excessive looping, especially when the second foreach isn’t required as $entry_id = $entry_cats[ $entry[‘entry_id’] ]; would be sufficient.

Although we could create a custom category field which will contain the translated category data we require and this will then be pulled in a utilised natively by Low Search indexing, we still need the extension for the normal entry data translations, so still need the category data added - back to square one.

There’s no mad rush for this as the language releases aren’t imminent, but would be good to be resolved at some stage in the future.

Thanks for your help.

Peter

#14

BoldMinded (Brian)

Good catch with the entry_ids, and yeah, I wasn’t really thinking about performance and was only testing with 20-30 entries. I did just push a version to the develop branch that does not query draft data, but assigns the open version to the draft as well for indexing purposes. It also includes your entry_ids fix. Try it out and see how it does?

#15

Peter Lewis

Hi Brian,

Sorry for the delay, yep that resolves the issue and doesn’t cause a memory issue, although we’ve only tested it on the 227 entries, but the next batch will be 10,000 entries, which could be interesting…

Thanks,

Peter

#16

Peter Lewis

So you’re aware we seem to be getting consistent issues when disabling the extension, resulting in errors:

A Database Error Occurred
Error Number: 1054
Unknown column 'publisher_status' in 'where clause'
DELETE FROM `exp_low_search_indexes` WHERE `publisher_status` != 'open' AND `publisher_lang_id` != '1'
Filename: /......./ext.publisher_low_search.php
Line Number: 439

This only occurs when the table has values in it. The problem then is that the publisher_status column i removed, and publisher_lang_id remains resulting in more errors and unable to re-enable the module without editing the database directly removing the offending columns and the primary key which uses the lang_id column.

#17

Peter Lewis

To clarify - to replicate: enable extension, re-index, disable extension.

#18

BoldMinded (Brian)

I’ll take a look early next week, kind of busy this week.

#19

BoldMinded (Brian)

Peter, I made some updates to the uninstall process. I don’t know why you’re trying to uninstall it, index, then re-install it.

As for the 10k entries… I just don’t know if adding support for that is reasonable for a free extension. That would require some queuing and a much more robust script.

#20

Peter Lewis

Because we were testing if the indexing was working, that’s why we were following that process, to then compare native index with and without the extension installed - appreciate it’s a fringe bug, but thought you’d prefer to know about it.

No worries on the bulk entries, I was just highlighting that I expect it to be an issue, but fully appreciate that this is a free add-on and wasn’t expecting anything to be done, if anything we’ll review and amend the code and share back any changes if required.

Many thanks for a your help - and for providing the add-on! 😊

#21

BoldMinded (Brian)

Going to close this ticket as the issue is resolved for builds with a reasonable number of entries.