Verify that a request is from Bing crawlers using Bing's DNS verification steps
You may wish to verify that a web crawler accessing your server is Bingbot (or another Bing user-agent) and not spammers or other bots scraping your site while claiming to be Bingbot. Since you cannot rely on the User-Agent
header which is easily spoofed, you need to use DNS look up to verify that the IP address belongs to Bing.
This library implements Bing's own verification steps outlined here.
npm install --save is-bing
const isBing = require('is-bing')
let ip = '157.55.33.18'
isBing(ip).then((outcome) => {
if (outcome) {
// it's bing.
}
}).catch(console.error)
app.enable('trust proxy')
app.use((req, res, next) => {
let ip = req.ip || req.connection.remoteAddress
isBing(ip).then(outcome => {
if (outcome) {
res.status(404).text('Nothing to scan') // block bing crawler
} else {
next() // it's a user
}
})
})
npm test
MIT
Rocco Musolino @roccomuso