This is an addition to the post http://www.wickedfire.com/design-de...ping-javascript-based-web-sites-php-curl.html. This post though I want to focus more on Fiddler2, since this tool is an immensely useful and free.
Again, like the last post, this post won't blow your mind if you already have a good understanding of scraping, packet sniffing and coding.
While fiddler2 is a good packet sniffer, it's actually far more than just that. It can do real-time packet modification and/or filtering, break points can be set for conditions and it can be easily extended with plug-ins written in C# (or any dotnet language), to name just a few of its features.
One of the most useful features is the 'Request Builder' which allows request to be easily built using either pre-excising data (previously sniffed packets) or hand crafted data. This allows request to be quickly modified to test responses from web servers.
Using Google Insights as an example, Google gives you the ability to download the results as a CSV (tab delimiter) if you are logged in.
The URL for the request has, among others, the parameter 'export=2'. Using Request Builder and changing the value to 1 Google will return the data as comma delimitated, something Google doesn't give you the option of doing normally.
Request Builder can also be used to test what headers the site is checking for to consider the request legitimate. For instance, testing if the server will still return valid data without a user-agent or missing cookies.
While this can be done with other methods (PHP etc), none of them, in my opinion, compare to the speed and ease of Fiddler2.
Again, like the last post, this post won't blow your mind if you already have a good understanding of scraping, packet sniffing and coding.
While fiddler2 is a good packet sniffer, it's actually far more than just that. It can do real-time packet modification and/or filtering, break points can be set for conditions and it can be easily extended with plug-ins written in C# (or any dotnet language), to name just a few of its features.
One of the most useful features is the 'Request Builder' which allows request to be easily built using either pre-excising data (previously sniffed packets) or hand crafted data. This allows request to be quickly modified to test responses from web servers.
Using Google Insights as an example, Google gives you the ability to download the results as a CSV (tab delimiter) if you are logged in.
The URL for the request has, among others, the parameter 'export=2'. Using Request Builder and changing the value to 1 Google will return the data as comma delimitated, something Google doesn't give you the option of doing normally.
Request Builder can also be used to test what headers the site is checking for to consider the request legitimate. For instance, testing if the server will still return valid data without a user-agent or missing cookies.
While this can be done with other methods (PHP etc), none of them, in my opinion, compare to the speed and ease of Fiddler2.
