Tab class#
- class Tab(websocket_url, target, browser=None, **kwargs)[source]#
Tab class is the controlling mechanism/connection to a ‘target’, for most of us ‘target’ can be read as ‘tab’. however it could also be an iframe, serviceworker or background script for example, although there isn’t much to control for those.
if you open a new window by using
browser.get(..., new_window=True)()your url will open a new window. this window is a ‘tab’. When you browse to another page, the tab will be the same (it is an browser view).So it’s important to keep some reference to tab objects, in case you’re done interacting with elements and want to operate on the page level again.
Custom CDP commands#
Tab object provide many useful and often-used methods. It is also possible to utilize the included cdp classes to to something totally custom.
the cdp package is a set of so-called “domains” with each having methods, events and types. to send a cdp method, for example
cdp.page.navigate, you’ll have to check whether the method accepts any parameters and whether they are required or not.you can use
`python await tab.send(cdp.page.navigate(url='https://yoururlhere')) `so tab.send() accepts a generator object, which is created by calling a cdp method. this way you can build very detailed and customized commands. (note: finding correct command combo’s can be a time consuming task, luckily i added a whole bunch of useful methods, preferably having the same api’s or lookalikes, as in selenium)
some useful, often needed and simply required methods#
find()| find(text)#find and returns a single element by text match. by default returns the first element found. much more powerful is the best_match flag, although also much more expensive. when no match is found, it will retry for <timeout> seconds (default: 10), so this is also suitable to use as wait condition.
find()| find(text, best_match=True) or find(text, True)#Much more powerful (and expensive!!) than the above, is the use of the find(text, best_match=True) flag. It will still return 1 element, but when multiple matches are found, picks the one having the most similar text length. How would that help? For example, you search for “login”, you’d probably want the “login” button element, and not thousands of scripts,meta,headings which happens to contain a string of “login”.
when no match is found, it will retry for <timeout> seconds (default: 10), so this is also suitable to use as wait condition.
select()| select(selector)#find and returns a single element by css selector match. when no match is found, it will retry for <timeout> seconds (default: 10), so this is also suitable to use as wait condition.
select_all()| select_all(selector)#find and returns all elements by css selector match. when no match is found, it will retry for <timeout> seconds (default: 10), so this is also suitable to use as wait condition.
await
Tab#calling await tab will do a lot of stuff under the hood, and ensures all references are up to date. also it allows for the script to “breathe”, as it is oftentime faster than your browser or webpage. So whenever you get stuck and things crashes or element could not be found, you should probably let it “breathe” by calling await page and/or await page.sleep()
also, it’s ensuring
urlwill be updated to the most recent one, which is quite important in some other methods.attempts to find the location of given template image in the current viewport the only real use case for this is bot-detection systems. you can find for example the location of a ‘verify’-checkbox, which are hidden from dom using shadow-root’s or workers.
await
Tab.template_location(and awaitTab.verify_cf)#attempts to find the location of given template image in the current viewport. the only real use case for this is bot-detection systems. you can find, for example the location of a ‘verify’-checkbox, which are hidden from dom using shadow-root’s or/or workers and cannot be controlled by normal methods.
template_image can be custom (for example your language, included is english only), but you need to create the template image yourself, which is just a cropped image of the area, see example image, where the target is exactly in the center. template_image can be custom (for example your language), but you need to create the template image yourself, where the target is exactly in the center.
example (111x71)#
this includes the white space on the left, to make the box center
Using other and custom CDP commands#
using the included cdp module, you can easily craft commands, which will always return an generator object. this generator object can be easily sent to the
send()method.send()#this is probably THE most important method, although you won’t ever call it, unless you want to go really custom. the send method accepts a
cdpcommand. Each of which can be found in the cdp section.when you import * from this package, cdp will be in your namespace, and contains all domains/actions/events you can act upon.
- add_handler(event_type_or_domain, handler)#
add a handler for given event
if event_type_or_domain is a module instead of a type, it will find all available events and add the handler.
if you want to receive event updates (network traffic are also ‘events’) you can add handlers for those events. handlers can be regular callback functions or async coroutine functions (and also just lamba’s). for example, you want to check the network traffic:
page.add_handler(cdp.network.RequestWillBeSent, lambda event: print('network event => %s' % event.request))
the next time you make network traffic you will see your console print like crazy.
- async bypass_insecure_connection_warning()[source]#
when you enter a site where the certificate is invalid you get a warning. call this function to “proceed” :return: :rtype:
- property closed#
- async connect(**kw)#
opens the websocket connection. should not be called manually by users :type kw: :param kw: :return:
- async disconnect()#
closes the websocket connection. should not be called manually by users.
- async evaluate(expression, await_promise=False, return_by_value=False)[source]#
- Return type:
Union[str,Any,Tuple[RemoteObject,Optional[ExceptionDetails]]]
- async find(text, best_match=True, return_enclosing_element=True, timeout=10)[source]#
find single element by text can also be used to wait for such element to appear.
- Parameters:
text (str) – text to search for. note: script contents are also considered text
best_match (
bool) –- param best_match:
when True (default), it will return the element which has the most comparable string length. this could help tremendously, when for example you search for “login”, you’d probably want the login button element, and not thousands of scripts,meta,headings containing a string of “login”. When False, it will return naively just the first match (but is way faster).
- type best_match:
bool
timeout (float,int) – raise timeout exception when after this many seconds nothing is found.
- async find_all(text, timeout=10)[source]#
find multiple elements by text can also be used to wait for such element to appear.
- async find_element_by_text(text, best_match=False, return_enclosing_element=True)[source]#
finds and returns the first element containing <text>, or best match
- Parameters:
text (
str) –best_match (bool) – when True, which is MUCH more expensive (thus much slower), will find the closest match based on length. this could help tremendously, when for example you search for “login”, you’d probably want the login button element, and not thousands of scripts,meta,headings containing a string of “login”.
- Returns:
- Return type:
- async find_elements_by_text(text, tag_hint=None)[source]#
returns element which match the given text. returns element which match the given text. please note: this may (or will) also return any other element (like inline scripts), which happen to contain that text.
- async get(url='chrome://welcome', new_tab=False, new_window=False)[source]#
top level get. utilizes the first tab to retrieve given url.
convenience function known from selenium. this function handles waits/sleeps and detects when DOM events fired, so it’s the safest way of navigating.
- async get_all_linked_sources()[source]#
get all elements of tag: link, a, img, scripts meta, video, audio
- async get_all_urls(absolute=True)[source]#
convenience function, which returns all links (a,link,img,script,meta)
- async get_frame_resource_tree()[source]#
retrieves the frame resource tree for current tab. There seems no real difference between Tab.get_frame_tree() but still it returns a different object :return: :rtype:
- async get_frame_tree()[source]#
retrieves the frame tree for current tab There seems no real difference between Tab.get_frame_resource_tree() :return: :rtype:
- async get_local_storage()[source]#
get local storage items as dict of strings (careful!, proper deserialization needs to be done if needed)
- Returns:
- Return type:
- property inspector_url#
get the inspector url. this url can be used in another browser to show you the devtools interface for current tab. useful for debugging (and headless) :return: :rtype:
- async js_dumps(obj_name, return_by_value=True)[source]#
dump given js object with its properties and values as a dict
note: complex objects might not be serializable, therefore this method is not a “source of thruth”
- Parameters:
- Return type:
Example
x = await self.js_dumps(‘window’) print(x)
‘…{ ‘pageYOffset’: 0, ‘visualViewport’: {}, ‘screenX’: 10, ‘screenY’: 10, ‘outerWidth’: 1050, ‘outerHeight’: 832, ‘devicePixelRatio’: 1, ‘screenLeft’: 10, ‘screenTop’: 10, ‘styleMedia’: {}, ‘onsearch’: None, ‘isSecureContext’: True, ‘trustedTypes’: {}, ‘performance’: {‘timeOrigin’: 1707823094767.9, ‘timing’: {‘connectStart’: 0, ‘navigationStart’: 1707823094768, ]… ‘
- async mouse_click(x, y, button='left', buttons=1, modifiers=0, _until_event=None)[source]#
native click on position x,y :type y:
float:param y: :type y: :type x:float:param x: :type x: :type button:str:param button: str (default = “left”) :type buttons:Optional[int] :param buttons: which button (default 1 = left) :type modifiers:Optional[int] :param modifiers: (Optional) Bit field representing pressed modifier keys.Alt=1, Ctrl=2, Meta/Command=4, Shift=8 (default: 0).
- async mouse_drag(source_point, dest_point, relative=False, steps=1)[source]#
drag mouse from one point to another. holding button pressed you are probably looking for
element.Element.mouse_drag()method. where you can drag on the element
- async open_external_inspector()[source]#
opens the system’s browser containing the devtools inspector page for this tab. could be handy, especially to debug in headless mode.
- async query_selector(selector, _node=None)[source]#
find single element based on css selector string
- Parameters:
selector (str) – css selector(s)
- Returns:
- Return type:
- async query_selector_all(selector, _node=None)[source]#
equivalent of javascripts document.querySelectorAll. this is considered one of the main methods to use in this package.
it returns all matching
nodriver.Elementobjects.
- remove_handler(event_type_or_domain, handler=None)#
remove a handler for given event :type event_type_or_domain:
Union[type,ModuleType,List[type]] :param event_type_or_domain: :type event_type_or_domain: :type handler:Union[Callable,Awaitable] :param handler: :type handler:
- async save_screenshot(filename='auto', format='jpeg', full_page=False)[source]#
Saves a screenshot of the page. This is not the same as
Element.save_screenshot, which saves a screenshot of a single element only
- async scroll_bottom_reached()[source]#
returns True if scroll is at the bottom of the page handy when you need to scroll over paginated pages of different lengths :return: :rtype:
- async scroll_down(amount=25)[source]#
scrolls down maybe
- Parameters:
amount (int) – number in percentage. 25 is a quarter of page, 50 half, and 1000 is 10x the page
- Returns:
- Return type:
- async scroll_up(amount=25)[source]#
scrolls up maybe
- Parameters:
amount (int) – number in percentage. 25 is a quarter of page, 50 half, and 1000 is 10x the page
- Returns:
- Return type:
- async select(selector, timeout=10)[source]#
find single element by css selector. can also be used to wait for such element to appear.
- async select_all(selector, timeout=10, include_frames=False)[source]#
find multiple elements by css selector. can also be used to wait for such element to appear.
- async send(cdp_obj, _is_update=False)#
send a protocol command. the commands are made using any of the cdp.<domain>.<method>()’s and is used to send custom cdp commands as well.
- async set_download_path(path)[source]#
sets the download path and allows downloads this is required for any download function to work (well not entirely, since when unset we set a default folder)
- async set_local_storage(items)[source]#
set local storage. dict items must be strings. simple types will be converted to strings automatically.
- async set_window_size(left=0, top=0, width=1280, height=1024)[source]#
set window size and position
- Parameters:
left – pixels from the left of the screen to the window top-left corner
top – pixels from the top of the screen to the window top-left corner
width – width of the window in pixels
height – height of the window in pixels
- Returns:
- Return type:
- async set_window_state(left=0, top=0, width=1280, height=720, state='normal')[source]#
sets the window size or state.
for state you can provide the full name like minimized, maximized, normal, fullscreen, or something which leads to either of those, like min, mini, mi, max, ma, maxi, full, fu, no, nor in case state is set other than “normal”, the left, top, width, and height are ignored.
- property target: TargetInfo#
- async template_location(template_image=None)[source]#
attempts to find the location of given template image in the current viewport the only real use case for this is bot-detection systems. you can find for example the location of a ‘verify’-checkbox, which are hidden from dom using shadow-root’s or workers.
template_image can be custom (for example your language, included is english only), but you need to create the template image yourself, which is just a cropped image of the area, see example image, where the target is exactly in the center. template_image can be custom (for example your language), but you need to create the template image yourself, where the target is exactly in the center.
example (111x71)#
this includes the white space on the left, to make the box center
- async verify_cf(template_image=None, flash=False)[source]#
convenience function to verify cf checkbox
template_image can be custom (for example your language, included is english only), but you need to create the template image yourself, which is just a cropped image of the area, see example image, where the target is exactly in the center.
example (111x71)#
this includes the white space on the left, to make the box center
- type template_image:
- param template_image:
template_image can be custom (for example your language, included is english only), but you need to create the template image yourself, which is just a cropped image of the area, where the target is exactly in the center. see example on (https://ultrafunkamsterdam.github.io/nodriver/nodriver/classes/tab.html#example-111x71),
- type template_image:
- type flash:
- param flash:
whether to show an indicator where the mouse is clicking.
- type flash:
- return:
- rtype:
- async wait_for(selector='', text='', timeout=10)[source]#
variant on query_selector_all and find_elements_by_text this variant takes either selector or text, and will block until the requested element(s) are found.
it will block for a maximum of <timeout> seconds, after which an TimeoutError will be raised
- property websocket: ClientConnection#
- async xpath(xpath, timeout=2.5)[source]#
find elements by xpath string. if not immediately found, retries are attempted until timeout is reached (default 2.5 seconds). in case nothing is found, it returns an empty list. It will not raise. this timeout mechanism helps when relying on some element to appear before continuing your script.
# find all the inline scripts (script elements without src attribute ) await tab.xpath('//script[not(@src)]') # or here, more complex, but my personal favorite to case-insensitive text search await tab.xpath('//text()[ contains( translate(., "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"),"test")]')
:return:List[nodriver.Element] or [] :rtype: