CLI Crawling in JoomSitemap

The JoomSitemap CLI plugin enables you to run sitemap crawls, generate XML sitemaps, and notify search engines directly from the command line. This is particularly useful for:

• Automated Scheduling: Set up periodic crawls using cron (Linux/Unix) or Task Scheduler (Windows)
• Server Resource Management: Run crawls during off-peak hours to minimize server load
• CI/CD Integration: Incorporate sitemap generation into deployment workflows
• Headless Environments: Manage sitemaps on servers without web interface access

Enable the Plugin

The plugin is disabled by default after installation:

1. Go to System → Plugins
2. Search for "JoomSitemap" or filter by "console" folder
3. Find CLI - JoomSitemap in the list
4. Click to enable the plugin
5. Save your changes

You can verify the plugin status in Components → JoomSitemap → CLI Crawling.

Available Commands

All JoomSitemap CLI commands use the Joomla CLI framework and share the joomsitemap: prefix.

crawl-website

Crawls your website to discover URLs and update the sitemap index, just like the backend crawler.

Basic Usage:

php cli/joomla.php joomsitemap:crawl-website

Command Options

Option	Description
--write-sitemap	Automatically write the XML sitemap files to disk after crawling
--notify-indexnow	Send notifications to IndexNow API when changes are detected (requires JoomSitemap IndexNow)
--debug	Enable debug mode - creates a detailed protocol without updating the index
--quiet or -q	Suppress non-error output during execution
--help	Display command help and available options

Examples

Basic crawl:

php cli/joomla.php joomsitemap:crawl-website

Crawl and generate sitemap:

php cli/joomla.php joomsitemap:crawl-website --write-sitemap

Crawl, generate sitemap, and notify IndexNow:

php cli/joomla.php joomsitemap:crawl-website --write-sitemap --notify-indexnow

Debug mode (no index changes):

php cli/joomla.php joomsitemap:crawl-website --debug

Silent execution (for cron jobs):

php cli/joomla.php joomsitemap:crawl-website --write-sitemap --quiet

Scheduling Periodic Crawls

Linux/Unix - Using Cron

Finding Your PHP CLI Path

which php

This typically returns /usr/bin/php or /usr/local/bin/php.

Setting Up the Cron Job

1. Open your crontab for editing:

   crontab -e

2. Add a cron entry (minute, hour, day, month, weekday, command)

Cron Examples

Daily at 11:55 PM:

55 23 * * * /usr/bin/php /var/www/html/cli/joomla.php joomsitemap:crawl-website --write-sitemap

Every 6 hours:

0 */6 * * * /usr/bin/php /var/www/html/cli/joomla.php joomsitemap:crawl-website --write-sitemap

Weekly on Sundays at 2:00 AM:

0 2 * * 0 /usr/bin/php /var/www/html/cli/joomla.php joomsitemap:crawl-website --write-sitemap

Daily at 11:55 PM with quiet output:

55 23 * * * /usr/bin/php /var/www/html/cli/joomla.php joomsitemap:crawl-website --write-sitemap --quiet

Important Notes:

• Replace PHP path with your server’s path
• Replace Joomla root path
• Use --quiet in cron
• Cron runs under its user permissions

Windows - Using Task Scheduler

Steps to add a scheduled crawl using Windows Task Scheduler.

Finding PHP Path

where php

Common locations:

• C:\php\php.exe
• C:\xampp\php\php.exe
• C:\wamp64\bin\php\php8.x\php.exe
• C:\laragon\bin\php\php-8.x\php.exe

Web Hosting Control Panel

Instructions for cPanel, Plesk, DirectAdmin and others.

cPanel Example

/usr/bin/php /home/username/public_html/cli/joomla.php joomsitemap:crawl-website --write-sitemap

Plesk Example

/usr/bin/php /var/www/vhosts/yourdomain.com/httpdocs/cli/joomla.php joomsitemap:crawl-website --write-sitemap

Monitoring and Troubleshooting

Includes issues, solutions, debug mode, checking cron logs, etc.

• Command not found: wrong PHP path
• Permission denied: fix ownership/permissions
• Timeout: increase max_execution_time
• Out of memory: raise memory_limit
• Cron not running: check logs, test manually

Advanced Usage

Logging to File

php cli/joomla.php joomsitemap:crawl-website --write-sitemap >> /var/log/joomsitemap.log 2>&1

Email Notifications

php cli/joomla.php joomsitemap:crawl-website --write-sitemap | mail -s "Sitemap Crawl Complete" This email address is being protected from spambots. You need JavaScript enabled to view it.

Multiple Sites Script

#!/bin/bash
sites=(
  "/var/www/site1.com"
  "/var/www/site2.com"
  "/var/www/site3.com"
)
for site in "${sites[@]}"
do
  echo "Crawling $site..."
  /usr/bin/php "$site/cli/joomla.php" joomsitemap:crawl-website --write-sitemap --quiet
done
echo "All sites crawled successfully"

Make executable and add to cron:

chmod +x /path/to/crawl-all-sites.sh

# In crontab:

0 2 * * * /path/to/crawl-all-sites.sh