Using a simple shell script to notify changes in a website
Malaysia’s controversial Anti-Fake News law was rushed through parliament and the senate, just before the general elections were announced.
Once passed in parliament and senate, it then requires approval from the current Agong or King of Malaysia before it is gazetted and published by the Attorney General Chambers Federal Gazette within 30 days.
When will it be published though? If we know the name of the bill we can write a simple shell script to check and email us when it is published online.
(Of course, it would be nice to have a way to subscribe to updates on the website, but that’s another issue)
The job
Since we know the name of the law, and the page where it will be published, all we need is to a simple script that checks say, every hour for when the name of the law starts appearing. We then need to email a notice only when it happens.
The tools
The following tools are standard or can easily be installed on Unix or Unix like systems such as FreeBSD, MacOS and Linux
- curl(1) program that can fetch contents for a given url
- grep(1) program that checks for a pattern
- mail(1) program that sends mail
- crontab(5) configuration file for service that runs programs at specific times
The script
#!/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/home/kaeru/binif curl --silent http://www.federalgazette.agc.gov.my/eng_main/main_akta.php?jenis_akta=Baru | grep -i Fake; then
echo “Sir, It has been published on AGC Website — AIDA”| mail -s “Fake News Gazetted” -c team@sinarproject.org khairil.yusof@sinarproject.org
fi
curl silent suppresses the progress update output and just sends the html contents of the url to the stdout (screen default), but we pipe it | to the grep command
grep command than processes the html to look for the keyword “fake”, -i makes it case insensitive
grep returns status 0 if found and 1 if not
if true, which means it was published, we then use the mail(1) program to send a simple message with subject to addresses to be notified
We then save this somewhere and mark it as executable ie chmod 700 scriptname
Scheduling it
All users can edit their own crontab schedule of entries. The following will run the script to check the website every hour at half past the hour.
30 * * * * /path/to/scriptMore reading
- IBM’s LPI 101 basics series of tutorials on Linux/Unix commands and tools
- Configuring Cron (FreeBSD)
- Unix Power Tools Third Edition
Learning Unix shell tools is an invaluable skill to have to be able to do a variety of random tasks.
