Covert Crawling: A Wolf Among Lambs (Break It!) presented at ShmooCon 2006

by Acidus ,

Tags: Security Web

Summary : Web application IDS evasion techniques and countermeasures is a mature area of study. LibWhisker-based apps and Snort have been in a tug-of-war for years. However, the initial reconnaissance of a website or web app has been largely neglected. Its either done by hand (which is tedious) or with a traditional crawler like wget (which is very noisy). An automated crawl appears as an enormous spike in hit count and byte transfer that is well outside the bell-curve for normal users.
This presentation will discuss theories and methods to hide an intelligent automated crawl of a target website or application inside the buzz of normal user activity. Some techniques include:
Spreading crawl across multiple IPs and time.
Following paths to links -vs- deep links.
Throttling crawl based on publicly available traffic stats and IP fragment ids.
Dynamic creation of fake Google referrers to a deep linked pages based on content of that page
Intelligent selection of proxies based on target country and website type.
Randomized link selection and overlap
Filtering of link targets based on popularity.
Intentional Traffic escalation (Slash-bombing)
This covert crawl will identify a subset of likely vulnerable pages that can later be attacked using IDS evasion techniques. You attacking fewer pages, and there is no advanced warning that an attack is eminent.
Code for a covert crawler implementing these techniques will be released.