Using a simple shell script to notify changes in a website

Khairil Yusof
2 min readApr 11, 2018
Malaysian Attorney General Chambers — e-Federal Gazette website

Malaysia’s controversial Anti-Fake News law was rushed through parliament and the senate, just before the general elections were announced.

Once passed in parliament and senate, it then requires approval from the current Agong or King of Malaysia before it is gazetted and published by the Attorney General Chambers Federal Gazette within 30 days.

When will it be published though? If we know the name of the bill we can write a simple shell script to check and email us when it is published online.

(Of course, it would be nice to have a way to subscribe to updates on the website, but that’s another issue)

The job

Since we know the name of the law, and the page where it will be published, all we need is to a simple script that checks say, every hour for when the name of the law starts appearing. We then need to email a notice only when it happens.

The tools

The following tools are standard or can easily be installed on Unix or Unix like systems such as FreeBSD, MacOS and Linux

  • curl(1) program that can fetch contents for a given url
  • grep(1) program that checks for a pattern
  • mail(1) program that sends mail
  • crontab(5) configuration file for service that runs programs at specific times

The script

#!/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/home/kaeru/bin
if curl --silent http://www.federalgazette.agc.gov.my/eng_main/main_akta.php?jenis_akta=Baru | grep -i Fake; then
echo “Sir, It has been published on AGC Website — AIDA”| mail -s “Fake News Gazetted” -c team@sinarproject.org khairil.yusof@sinarproject.org
fi

curl silent suppresses the progress update output and just sends the html contents of the url to the stdout (screen default), but we pipe it | to the grep command

grep command than processes the html to look for the keyword “fake”, -i makes it case insensitive

grep returns status 0 if found and 1 if not

if true, which means it was published, we then use the mail(1) program to send a simple message with subject to addresses to be notified

We then save this somewhere and mark it as executable ie chmod 700 scriptname

Scheduling it

All users can edit their own crontab schedule of entries. The following will run the script to check the website every hour at half past the hour.

30 * * * *  /path/to/script

More reading

Learning Unix shell tools is an invaluable skill to have to be able to do a variety of random tasks.

--

--