Allow Viber share links and Ahrfes site crawling

By default GridPane has blocked viber and ahrefs bots. We have to made some changes to the servers in order to allow these bots to communicate with our sites.1

Step 1

Edit this file: nano /etc/nginx/extra.d/allow-bots-7g-bad-bot-context.conf

The main concept here is to remove from these lines all the bots that we don’t want to have. So just copy and paste the following code:

"~*(acapbot|acoonbot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|exabot|extract|eyenetie|feedfinder|flaming|flashget|flicky|foobot|g00g1e|getright|gigabot|go-ahead-got|gozilla|grabnet|grafula|harvest|heritrix|httrack|icarus6j|jetbot|jetcar|jikespider|kmccrew|leechftp|libweb|linkextractor|linkscan|linkwalker|masscan|miner|mechanize|morfeus|moveoverbot|netmechanic|netspider|nicerspro|nikto|ninja|nutch|octopus|pagegrabber|petalbot|planetwork|postrank|proximic|purebot|pycurl|python|queryn|queryseeker|radian6|radiation|realdownload|scooter|seekerspider|semalt|siclab|sindice|sistrix|sitebot|siteexplorer|sitesnagger|skygrid|smartdownload|snoopy|sosospider|spankbot|spbot|sqlmap|stackrambler|stripper|sucker|surftbot|sux0r|suzukacz|suzuran|takeout|teleport|telesoft|true_robots|turingos|turnit|vampire|vikspider|voideye|webleacher|webreaper|webstripper|webvac|webviewer|webwhacker|winhttp|wwwoffle|woxbot|xaldon|xxxyy|yamanalab|yioopbot|youda|zeus|zmeu|zune|zyborg)" 3000;
"~*(alexibot|majestic|mj12bot|rogerbot)" 2002;

Step 2

Also edit this file: nano /etc/nginx/extra.d/bad-bot-whitelist-7g-context.conf

Just copy and paste the following code

# Remove Rule #5 for $bad_bot_7g
set $exclusion_rule_match "";
if ($bad_bot_7g) {
    set $exclusion_rule_match 5;
    set $exclusion_rule_match_ahrefs 2001;
}
if ($bad_bot_7g = $exclusion_rule_match) {
   set $7g_drop_bad_bot 0;
}

# Remove Rule 2001 for $bad_bot_7g
if ($bad_bot_7g = $exclusion_rule_match_ahrefs) {
   set $7g_drop_bad_bot 0;
}

Step 3

Check NGINX syntax with the following command: nginx -t

If everything is correct you should be able to see the following message:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

Step 4

Reload NGINX with the following command: gp ngx reload

https://www.youtube.com/watch?v=TThpz9WfjD4

⚠️ Important step

if you are edit the file and you are seeing > in the end of the file this means the line is turncated.

"~*(acapbot|acoonbot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumb>

To resolve that and see the full line you have to run the file with cat command. Then you will be able to copy/paste the whole line.

cd /etc/nginx/common
cat 7g-mappings.conf