Site wide Search in Laravel with Scout

Watch this tutorial on Youtube:

The “Search everything” box in the navigation bar is a very common feature in modern web app. I like it a lot, but implementing it in our own web app could be challenge. Here’s my solution to tackle this problem.

Prerequisite

  1. Laravel Scout (I’m using TNTSearch driver here)
  2. Intermediate knowledge of Laravel

Note: I’m using Laravel 8 at the time of writing this article.

Full source code for this tutorial is available here. I will only cover the core implementation in this article. The source code is commented, please check it out if you need more details.

TLDR; If you don’t feel like reading / implementing your own, I’ve created a package for this.

The plan

Here’s a high-level overview on how are we going to achieve this.

  1. Create a global site search endpoint.
  2. Load all available models in the model folder.
  3. Loop through the models, and call the Laravel Scout’s ::search() function on each model.
  4. Each record found should include:
    a. match — the match found in our model records
    b. model — the related model name, eg Post , User
    c. view_link — the URL to navigate to the target resource page.
  5. Combine all results together and load them in an API Resource Collection and return to the front-end.

Show me the Code

Okay…okay, here we go. I assumed you have already installed Laravel Scout and TNTSearch at this point.

Set up an API routes to perform the full site search

// /routes/api.php
Route::get('/site-search', [\App\Http\Controllers\SitewideSearchController::class, 'search']);

Setting up the Models

I created 3 models for this tutorial. Here is the database diagram, aka Entity Relationship Diagram (ERD).

Each model class should use the Searchable trait, otherwise Laravel Scout will not work. You should overwrite the toSearchableArray() method to tell Scout which fields that you want to include in the full text search. Here is an example of my Post model.

Note: It is important to declare the searchable fields as a class constant, as we will be using it later.

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Laravel\Scout\Searchable;

class Post extends Model
{
use HasFactory, Searchable;

const SEARCHABLE_FIELDS = ['id', 'title', 'body'];

public function toSearchableArray()
{
return $this->only(self::SEARCHABLE_FIELDS);
}
}

Setting up the Site Wide search API

Here’s the code, enjoy and you are on your own from now on😛. (Just kidding, explanation is after the embedded code.)

Let’s break it down

I would suggest to have the source code on the side so you can follow along with the explanation. Medium’s code block can sometimes be hard to read.

First of all, the search function is the main function of this controller, and it is where the API request will hit.

We want to load all the models in our app. One way to do that is to use Laravel’s File facade. We just need to provide the directory path to it and it will magically return all the files back to us.

// ..
$files = File::allFiles(app()->basePath() . '/app/Models');

Next, in the map function, we will parse the Model name from the file name. Now we will have a collection of model names.

...->map(function (SplFileInfo $file){
$filename = $file->getRelativePathname();

// assume model name is equal to file name
/* making sure it is a php file*/
if (substr($filename, -4) !== '.php'){
return null;
}
// removing .php
return substr($filename, 0, -4);

})

However, we may have files other than the models living inside our Model directory, or models that didn’t implement the Searchable trait. We need to filter them out. We also want to exclude all the models listed inside the toExclude array.

We will call the filter function on our collection. We can make use of PHP’s ReflectionClass to retrieve class information.

...->filter(function (?string $classname) use($toExclude){
if($classname === null){
return false;
}

// using reflection class to obtain class info dynamically
$reflection = new \ReflectionClass($this->modelNamespacePrefix() . $classname);

// making sure the class extended eloquent model
$isModel = $reflection->isSubclassOf(Model::class);

// making sure the model implemented the searchable trait
$searchable = $reflection->hasMethod('search');

// filter model that has the searchable trait and not in exclude array
return $isModel && $searchable && !in_array($reflection->getName(), $toExclude, true);

})

Now we can ensure all the model names remaining are the ones that we want to conduct the full text search. Our goal here is to retrieve and combine the results together in 1 big collection.

We will map the model names. To improve the user’s search experience, we want to include the match , model and view_link on all search results. The idea is to send these results back to the frontend so the user can understand the search context via match and model . view_link will allow the user to navigate to the resource page in the front-end.

The following will perform a full text search against our model. We use the take function to limit the search result set to the first 5 matches, otherwise the performance will drop significantly if we have a squillion of records in our database.

$model::search($keyword)->take(5)->get() 

We will then map the results so each of them will contain match , model and view_link .

I’m using PHP 7.4’s arrow function here to filter out the id field. You can use the traditional anonymous function if you like.

To create the match , we first need to find out the position of our $keyword string from the database record. $serializedValues is the aggregation of all of our searchable field values.

$fields = array_filter($model::SEARCHABLE_FIELDS, fn($field) => $field !== 'id');

// only extracting the relevant fields from our model
$fieldsData = $modelRecord->only($fields);

// joining the fields together
$serializedValues = collect($fieldsData)->join(' ');

// finding the position of match
$searchPos = strpos(strtolower($serializedValues), strtolower($keyword));

Once we found the position, we will include the neighbouring text of our match. And depending on the position, ie far right or far left or middle of $serializedValues , we prepend and append ... on the match.

// the buffer number dictates how many neighbouring characters to display
$start = $searchPos - self::BUFFER;

// we don't want to go below 0 as the starting position
$start = $start < 0 ? 0 : $start;

// multiply 2 buffer to cover the text before and after the match
$length = strlen($keyword) + 2 * self::BUFFER;

// getting the match and neighbouring text
$sliced = substr($serializedValues, $start, $length);

// adding prefix and postfix dots

// if start position is negative, there is no need to prepend `...`
$shouldAddPrefix = $start > 0;
// if end position went over the total length, there is no need to append `...`
$shouldAddPostfix = ($start + $length) < strlen($serializedValues) ;

$sliced = $shouldAddPrefix ? '...' . $sliced : $sliced;
$sliced = $shouldAddPostfix ? $sliced . '...' : $sliced;

And finally, we add the attributes in the model record.

// use $slice as the match, otherwise if undefined we use the first 20 character of serialisedValues
$modelRecord->setAttribute('match', $sliced ?? substr($serializedValues, 0, 20) . '...');
// setting the model name
$modelRecord->setAttribute('model', $classname);
// setting the resource link
$modelRecord->setAttribute('view_link', $this->resolveModelViewLink($modelRecord));

The resolveModelViewLink is a helper function to generate the view_link for us. We can customise the model resource link in the $mapping array.

$mapping = [
\App\Models\Comment::class => '/comments/view/{id}'
];

// getting the Fully Qualified Class Name of model
$modelClass = get_class($model);

// converting model name to kebab case
$modelName = Str::plural(Arr::last(explode('\\', $modelClass)));
$modelName = Str::kebab(Str::camel($modelName));

// attempt to get from $mapping. We assume every entry has an `{id}` for us to replace
if(Arr::has($mapping, $modelClass)){
return str_replace('{id}', $model->id, $mapping[$modelClass]);
}
// assume /{model-name}/{model_id}
return URL::to('/' . strtolower($modelName) . '/' . $model->id);

We have a nested collection. The first level is from the map function on the model names. The second level is created from Scout’s search results, ie we are returning another map in the previous model names map. We need to flatten the collection.

...->flatten(1);

Finally, we wrap our collection in a SiteSearchResource class where we standardise the fields in our API response.

Here is how it looks like:

<?php

namespace App\Http\Resources;

use Illuminate\Http\Resources\Json\JsonResource;

class SiteSearchResource extends JsonResource
{
/**
* Transform the resource into an array.
*
* @param \Illuminate\Http\Request $request
* @return array
*/
public function toArray($request)
{
return [
'id' => $this->id,
'match' => $this->match,
'model' => $this->model,
'view_link' => $this->view_link,
];
}
}

And our sample API response:

Putting it all together

We can now build a search bar in our front-end that calls this endpoint that looks something like this (source code available in my demo repo):

Caveat

This solution has a few caveats and assumptions:

  1. This method tightly coupled the API server against the frontend Url. That means, if your frontend URL changes, you will need to change the corresponding API response as well.
  2. We assume all of our models live in app/Models/.
  3. We assume the model class name is the same as the model file name, by PSR-4 standard.
  4. Performance could be an issue if there are a lot of models.

That’s it! Happy coding 😎

Web Development. https://acadea.io/learn . Follow me on Youtube: https://www.youtube.com/c/acadeaio

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store