eyes.celery¶
eyes celery module
Eyes celery app module
eyes.celery.crawler.tasks¶
Eyes celery crawler tasks
- class eyes.celery.crawler.tasks.CrawlerTask¶
Crawler Base Task
- after_return(*args, **kwargs)¶
Callback after finishing a job
- property sess¶
Returns SQLAlchemy Session
- (task)eyes.celery.crawler.tasks.crawl_ptt_post(url: str, board: str) Union[Dict, NoneType]¶
Crawl a ptt post and store it into database
- Parameters
url (str) – post url
board (str) – board name
- Returns
post dictionary
- Return type
Optional[Dict]
- (task)eyes.celery.crawler.tasks.crawl_ptt_board_list(top_n: Union[int, NoneType] = None) Union[List[Dict], NoneType]¶
Crawl ptt board list
- Parameters
top_n (Optional[int]) – top N boards
- Returns
Optional[List[Dict]]
- (task)eyes.celery.crawler.tasks.crawl_dcard_post(post_id: int) Union[Dict, NoneType]¶
Crawl dcard post
- Parameters
post_id (int) – post id
- Returns
post dictionary
- Return type
Optional[Dict]
- (task)eyes.celery.crawler.tasks.crawl_dcard_board_list(top_n: Union[int, NoneType] = None) Union[List[Dict], NoneType]¶
Crawl dcard board list
- Parameters
top_n (Optional[int]) – top N boards
- Returns
Optional[List[Dict]]
- (task)eyes.celery.crawler.tasks.crawl_wiki_entity(url: str, label: int) Union[Dict, NoneType]¶
Crawl wiki entity
- Parameters
url (str) – entity url
label (int) – entity label id
- Returns
wiki entity dictionary
- Return type
Optional[Dict]
eyes.celery.stats.tasks¶
Eyes celery stats tasks
- class eyes.celery.stats.tasks.StatsTask¶
Stats Base Task
- after_return(*args, **kwargs)¶
Callback after finishing a job
- property config¶
Returns Eyes config
- property sess¶
Returns SQLAlchemy Session
- property nlp¶
Returns spacy language model
- (task)eyes.celery.stats.tasks.ptt_monthly_summary(year, month) Dict¶
PTT monthly summary
- Parameters
year (int) – year
month (int) – month
- (task)eyes.celery.stats.tasks.stats_entity_summary(year, month, limit=None) Dict¶
Entity stats
eyes.celery.ml.tasks¶
Eyes celery ml tasks
- class eyes.celery.ml.tasks.MlTask¶
Ml Base Task
- property nlp¶
Returns spacy language model
- property sess¶
Returns SQLAlchemy Session
- (task)eyes.celery.ml.tasks.transform_ptt_post_to_spacy_post(post_id: str) Dict¶
Transform ptt post to spacy post
- Parameters
post_id (str) – ptt post id
- Returns
transformed post
- Return type
Dict