gogogogo

[mattseh]

Trying out this Go thing... | Disqus: The Official Blog

oh my.

I've been playing with go a bit. It's worth investigation, but I'll stick with python for a lot of what I do.

goroutines + channels = magic.

 

[Rage9]

Jake what have you done?

 

[mattseh]

Not really a great showcase for go's unique features, but:

package main

import "bufio"
import "os"
import "fmt"
import "net/url"
import "strings"

func main() {
    domainsMap := make(map[string]bool)
    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        parsed, err := url.Parse(scanner.Text())
        if err == nil {
            domainsMap[strings.ToLower(parsed.Host)] = true
        }
    }    
    fmt.Println(len(domainsMap))
}

Accepts piped urls

 

[Mahzkrieg]

Can you explain what that code does? I'd like to see how I'd write it in Clojure.

 

[mattseh]

The scanner takes each line from piped in data (from cat / whatever), parses the url on each line, takes the domain, adds it as a key to a hash, counts how many keys in the hash.

 

[lwbco]

so its like " | wc -l" then ???

 

[mattseh]

so its like " | wc -l" then ???

no.

It works out how many unique domains there are in a list of urls.

len(set(urlparse.urlparse(url).netloc.lower() for url in urls))

 

[lwbco]

oh, i see, so like "| cut -d\/ -f1 | sort -u | wc -l", cool go-ry bro

#100th post

 

[yast]

oh, i see, so like "| cut -d\/ -f1 | sort -u | wc -l", cool go-ry bro

#100th post

bash ftw

 

[mattseh]

cat urls.txt | head -n 50 | cut -d\/ -f1 | sort -u
http:

 

[mattseh]

-f3 makes it work

 

[Jake232]

Bumping this back up because Go is awesome, and more people should try it out.

Here, have a spin function.

func spin(text_input string) string {
	rand.Seed(time.Now().UTC().UnixNano())

	// Common error is no ending }
	if strings.Count(text_input, "{")-strings.Count(text_input, "}") == 1 {
		text_input = text_input + "}"
	}

	count := strings.Count(text_input, "{")

	// Replace bitchez
	for i := 0; i < count; i++ {
		field := spinRegex.FindStringSubmatch(text_input)
		replacements := strings.Split(field[1], "|")
		text_input = strings.Replace(text_input, field[0], replacements[rand.Intn(len(replacements))], 1)
	}

	return text_input

}

 

[mattseh]

i implemented channels and goroutines in python, so cool story bro.

 

[Jake232]

i implemented channels and goroutines in python, so cool story bro.

Come back when you can thread bro (without locking the GIL).

 

[mattseh]

Come back when you can thread bro (without locking the GIL).

I'll just run a couple hundred processes over a few clouds and some dedis.

 

[medway]

Speaking of unique domains anyone got code in Python handy to delete duplicate ones? Using set() to delete dup urls but nothing for domains yet.

 

[Jake232]

Speaking of unique domains anyone got code in Python handy to delete duplicate ones? Using set() to delete dup urls but nothing for domains yet.

import urlparse

urls = set(open('links.txt', 'r').read().replace('\r\n').split('\n'))
seen_domains = []
output = []

for url in urls:
	domain = urlparse.urlparse(url).netloc.lower()
	if domain not in seen_domains:
		seen_domains.append(domain)
		output.append(url)

print "Found %s unique domains" % len(output)

f = open('output.txt', 'w')
for i in output:
	f.write(i+"\n")

Will keep one url from each domain. (untested, but should work)

 

[Jake232]

I'll just run a couple hundred processes over a few clouds and some dedis.

Or, you could use Go and run a couple hundred processes over a few clouds and some dedis, with each process threading.

 

[mattseh]

Speaking of unique domains anyone got code in Python handy to delete duplicate ones? Using set() to delete dup urls but nothing for domains yet.

domains = set(urlparse.urlparse(url).netloc.lower() for url in urls)

 

[mattseh]

Jake, in this house, we use join() to write a list to a file.