Craigslist Commandline Search Tool

THIS TOOL HAS BEEN MOVED HERE:
https://zentastic.me/blog/zenclsh/

One of the tasks that I do in maintaining the kit car blog is searching Craigslist for interesting finds. Problem is, there’s no good way to search for multiple terms, let alone multiple cities — so in a single day, I’d have about 1,200 searches to do. So I wrote this quick little bot to do it for me.

harvestcl.zip (30k)

This contains a few files:

  • harvestcl.exe – This is the actual executable. You should run it from the command line, although you can just click on it as well, as long as the support files are in the directory you’re running it from.
  • clcities.txt – This is a list of all the cities to search. Blank lines are ignored, as are lines beginning with “#” (so you can comment out cities by putting a “#” in front of them rather than deleting the line). The file is currently seeded with all cities in the US, Canada, the UK, Australia, and New Zealand.
  • clsearch.txt – This tells the program what searches you’d like to do. There should be one search per line, in the format “section,term”, where section is the short form name of the section as defined by Craigslist (ie. “sss” for all, “car” for cars and trucks, “clt” for collectibles, and so on), and the term is the search term. If the search term has a space in it, make sure you surround it by quotes. As with the cities file, you can comment out lines with the “#” character. The file currently contains the search that I do for the kit car site.
  • harvestcl.bas – For those that want it, this is the PB/CC 4 source code. Feel free to modify it or use parts of it for other projects — it includes some simple parsing code and http get code with support for 302 redirects (used here to redirect requests to country-specific Craigslist sites). This can be deleted.

When the program runs, it will search all the sites and create (and try and display) a file called harvestcl.html. Search matches will be listed by description (with a link) and split by the date they were posted. Every time you run it, new entries will be added to the same file, with the new ones being tagged with “NEW” so you don’t miss them. It maxes out at 15,000 matches, although you can change this in the code.

Hope this is useful for someone… If you’re looking for specialty items, it’s an effective tool that can be run daily to make sure you don’t miss something, no matter where in the world/country it is posted. Feel free to email me about this program at snowrail@gmail.com or post here if you have questions.

10 Comments

  1. Scott wrote:

    Thank you! This saves me so much time!!

    Wednesday, July 23, 2008 at 6:57 pm | Permalink
  2. Shannon wrote:

    Thanks. If there’s an interest in it, I may write a quick graphical front end of it so it’s not so “raw”… I’ve been using it a lot as well.

    Wednesday, July 23, 2008 at 7:14 pm | Permalink
  3. https://howto.wired.com/wiki/Search_Every_Craigslist_Site_at_Once

    Check that out, it’s probably not as good as what you posted, but it’s helped me!

    Wednesday, July 23, 2008 at 11:24 pm | Permalink
  4. Shannon wrote:

    That’s a good way to do it, but I found that Google does not completely index Craigslist — I discovered this when I was trying to re-find ads that I’d replied to by searching for text inside them. That method would however allow more complex searches which is good!

    Thursday, July 24, 2008 at 6:23 am | Permalink
  5. Danielle wrote:

    This will certainly come in handy. Thank you!

    Thursday, July 24, 2008 at 8:30 am | Permalink
  6. Scott wrote:

    Yea if it looked a little more user freindly to some people I think alot more people would use it. Ive show’d a few freinds but they find it abit confusing.

    Thursday, July 24, 2008 at 2:34 pm | Permalink
  7. Tyler wrote:

    Absolutely amazing!

    Thursday, July 24, 2008 at 2:41 pm | Permalink
  8. Thomas wrote:

    I like the one you wrote because I can bot that. However you could use this also

    https://www.craigshelper.com/

    Thursday, July 24, 2008 at 4:30 pm | Permalink
  9. Shannon wrote:

    I’d been using CraigsHelper until now — it’s great, but it’s not automated so you have to click through the cities one by one.

    Thursday, July 24, 2008 at 4:44 pm | Permalink
  10. Devan wrote:

    I guess I never thought of google not indexing everything. your tool does look badass, and easy to use.

    Sunday, July 27, 2008 at 9:26 am | Permalink