Skip to main content

bot hit - server response time and overall page loadtime increase - aws, ec2, google analytic, cloudflare, ms sql cpu usage 100%

 Dear All,

I've recently facing website page load time and server response time issue, 

troubleshooting - 

1. IIS log via - http log browser (tool to identify the bot hit)

2. disallow in robots.txt

3. article found and feature enabled in Cloudflare 

 Bot Fight Mode

Challenge requests matching patterns of known bots before they can access your site.

Requests matching Cloudflare-identified, non-legitimate automated traffic patterns will be challenged and/or blocked by Cloudflare.

And,

JavaScript Detections

Use lightweight, invisible JavaScript detections to improve Bot Management products.

4. AWS EC2 

Add new security group for Cloudflare ips list

Ref: https://dev.to/johnmccuk/automate-aws-security-group-with-cloudflare-ips-la6


And referred follow articles:

https://www.imperva.com/products/advanced-bot-protection-management/

https://blog.apnic.net/2019/11/15/how-to-hide-aws-ec2-instances-from-network-scanning-bots-using-ipv6/

https://pawelurbanek.com/ec2-ssh-dynamic-access





 

Comments

Post a Comment

Popular posts from this blog

IPv4 header to requests when a client is using IPv6

  What is Pseudo IPv4? As a stopgap to accelerate the adoption of IPv6, Cloudflare offers Pseudo IPv4 which supports IPv6 addresses in legacy applications expecting IPv4 addresses. The goal is to provide a nearly unique IPv4 address for each IPv6 address, using Class E IPv4 address space, which is designated as experimental and would not normally see traffic. To learn more see  here . Options Add header:  Add additional Cf-Pseudo-IPv4 header only Overwrite headers:  Overwrite the existing Cf-Connecting-IP and X-Forwarded-For headers with a pseudo IPv4 address Cloudflare
Post a Comment
Read more

Intrusion detection system

An   intrusion detection system   ( IDS ) is a device or   software application   that monitors network or system activities for malicious activities or policy violations and produces reports to a management station.  IDS come in a variety of “flavors” and approach the goal of detecting suspicious traffic in different ways. There are network based (NIDS) and host based (HIDS) intrusion detection systems. Some systems may attempt to stop an intrusion attempt but this is neither required nor expected of a monitoring system.  Intrusion detection and prevention systems (IDPS) are primarily focused on identifying possible incidents, logging information about them, and reporting attempts. In addition, organizations use IDPSes for other purposes, such as identifying problems with security policies, documenting existing threats and deterring individuals from violating security policies.  IDPSes have become a necessary addition to the security infrastru...
Post a Comment
Read more

Amazon AWS - Cloudfront configuration

Amazon Cloudfront Dynamic Content Delivery HTTP GET and HEAD   requests are currently supported by Amazon CloudFront. Over time, we will add support for POST requests.   Our sample setup You can convert your existing blog or setup a new blog from scratch to use cloudfront. For demonstration purpose, I'm going to setup a new blog called www.contentdeliverynetworklog.com and serve the whole blog via cloudfront as follows: Blog domain name:   www.contentdeliverynetworklog.com   (CNAME to cloudfront) Blog origin domain name:   cp.contentdeliverynetworklog.com . To fetch web content from this origin web server. Static asset domain name:   s0.contentdeliverynetworklog.net   (CNAME to cloudfront). Static asset origin domain name:   origin.contentdeliverynetworklog.net . Custom origin IP address for both static/dynamic assets:   75.126.153.203 Caching ruleset for your dynamic and static assets: I am going to use a combination of Batca...
8 comments
Read more