Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Specifying Headers in context.enqueue_links Method #894

Open
zhouya2010 opened this issue Jan 10, 2025 · 2 comments
Open
Assignees
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@zhouya2010
Copy link

zhouya2010 commented Jan 10, 2025

When using the context.enqueue_links method, there is currently no way to specify custom headers (e.g., cookies or other required headers) for the links being enqueued. Some websites require specific headers or cookies to properly retrieve data from detail pages. It would be helpful to add support for passing headers or request options to context.enqueue_links to handle such cases.

Example use case:

headers = {
    "Cookie": "session_id=abc123",
    "User-Agent": "Custom User Agent"
}
context.enqueue_links(selector="a.detail-page", headers=headers)

This feature would greatly enhance flexibility when crawling websites that require authentication or specific headers for access.

Thank you!

@github-actions github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Jan 10, 2025
@janbuchar
Copy link
Collaborator

I believe that adding a transform_request_function parameter to enqueue_links would help, am I correct? The JS version of Crawlee has it - https://crawlee.dev/api/core/interface/EnqueueLinksOptions#transformRequestFunction

@janbuchar janbuchar added the enhancement New feature or request. label Jan 10, 2025
@zhouya2010
Copy link
Author

Yes, that's right

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

3 participants