Botflow release v0.1.9 with Http Server support¶
Officially rename project to Botflow.
Enable Http Server support.Pipe can be work as Coroutine to integrate with other Asyncio framework.
from botflow import * from aiohttp import web p = Pipe( {"msg":"hello world!"} ) app = web.Application() app.add_routes([ web.get('/', p.aiohttp_json_handle) ]) Bot.run_app(app)
Add new Route “SendTo”. It can be used for redirect to data flow to the Node.
def filter_out(url): global count if 'http' not in url: url = "http://127.0.0.1:8080{}".format(url) if url in seen : #filter out processed links return None seen.add(url) return url def find_all_links(r): for a in r.soup.find_all('a', href=True): yield a.get('href') b = Return( filter_out, #filter out processed links HttpLoader(), find_all_links, #find all links in new page. ) Pipe( "http://127.0.0.1:8080/", b, SendTo(b), #send new url to process again . it will make a loop )
Add new Node type “SpeedLimit” “Delay” .For speed control
Add new Node type “Zip”. For zip multi flow item to list.
Rewrite whole project for code more readable.
Import flow performance by optimize graph generate algorithm .
Import crawler case performance. For local server broad crawler bench test. it can reach 760 pages per second. 10x faster than Scrapy.