I'm trying to move through different pages on a website called iens. I'm using selenium + python to click on "volgende" (which means "next" in dutch), but I want my program to keep on clicking on next until there are no more pages left using a while loop. So in this case I want my program to end at page 23. Right now I'm able to get to page 2, by closing the cookie pop up message, waiting until it's closed en then clicking on the "Volgende" button.
My code looks like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdrivermon.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_path = '/Users/user/Downloads/chromedriver'
driver = webdriver.Chrome(chrome_path)
driver.get('+utrecht')
#wait for cookie message
close_icon = WebDriverWait(driver, 5,
0.25).until(EC.visibility_of_element_located([By.CSS_SELECTOR,
'.cookiePolicy-close']))
close_icon.click()
#wait for cookie message to disappear
WebDriverWait(driver, 5,
0.25).until(EC.invisibility_of_element_located([By.CSS_SELECTOR,
'.cookiePolicy-close']))
click_icon = WebDriverWait(driver, 5,
0.25).until(EC.visibility_of_element_located([By.LINK_TEXT,
'Volgende']))
click_icon.click()
The website is called +utrecht
Thanks in advance!
I'm trying to move through different pages on a website called iens. I'm using selenium + python to click on "volgende" (which means "next" in dutch), but I want my program to keep on clicking on next until there are no more pages left using a while loop. So in this case I want my program to end at page 23. Right now I'm able to get to page 2, by closing the cookie pop up message, waiting until it's closed en then clicking on the "Volgende" button.
My code looks like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.mon.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_path = '/Users/user/Downloads/chromedriver'
driver = webdriver.Chrome(chrome_path)
driver.get('https://www.iens.nl/restaurant+utrecht')
#wait for cookie message
close_icon = WebDriverWait(driver, 5,
0.25).until(EC.visibility_of_element_located([By.CSS_SELECTOR,
'.cookiePolicy-close']))
close_icon.click()
#wait for cookie message to disappear
WebDriverWait(driver, 5,
0.25).until(EC.invisibility_of_element_located([By.CSS_SELECTOR,
'.cookiePolicy-close']))
click_icon = WebDriverWait(driver, 5,
0.25).until(EC.visibility_of_element_located([By.LINK_TEXT,
'Volgende']))
click_icon.click()
The website is called https://www.iens.nl/restaurant+utrecht
Thanks in advance!
Share Improve this question asked Dec 7, 2016 at 15:40 titusAdamtitusAdam 8091 gold badge17 silver badges38 bronze badges1 Answer
Reset to default 7Because of automatic scrolling each time you switch to next page, webdriver tries to click Next button, but some other element receives click. You can use this solution:
while True:
try:
click_icon = WebDriverWait(driver, 5, 0.25).until(EC.visibility_of_element_located([By.LINK_TEXT, 'Volgende']))
click_icon.click()
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'main:not([style*="margin-top"])')))
except:
break
Another simple solution (only if your goal is to reach last page) is to get last page without clicking Next all the time:
driver.find_elements_by_xpath('//div[@class="pagination"]/ul/li/a')[-2].click()