Skip to content

Move check_feed_listeners to async service #848

Description

@max-zilla

currently in backend/app/routes/files we have:

 # Add entry to the file index
    await index_file(es, FileOut(**new_file.dict()))

    # TODO - timing issue here, check_feed_listeners needs to happen asynchronously.
    time.sleep(1)

    # Submit file job to any qualifying feeds
    await check_feed_listeners(
        es,
        FileOut(**new_file.dict()),
        user,
        rabbitmq_client,
    )

Currently, check_feed_listeners performs queries on Elasticsearch to check if extractors should be triggered. It seems that the index_file() call takes a second to propagate and I was seeing situation where feed wasn't triggering after the index if I didn't delay for a second.

This should probably follow a different pattern:

  • index the file
  • issue a message to RMQ or a memory queue that says "you need to check this file for extractors"
  • a separate (different thread) service handles those message checks, similar to our heartbeat/messages services

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or request

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions