Automating cookie table generation with Selenium: A tester’s guide

Cookies are small text files that websites store on your device (computer, smartphone, tablet) when you visit them. They help websites remember your preferences, login information, and other details to enhance your browsing experience.

‍

A cookie table is a way to list and understand the cookies that a website uses. It presents details about each cookie in a structured format, making it easier to see what data is being stored and why.

Components of a cookie table

A typical cookie table includes the following columns:

1. Name: The name of the cookie. Each cookie has a unique name that identifies it.
2. Value: The data stored in the cookie. This could be a user ID, session token, or other information.
3. Domain: The domain (website address) that set the cookie. It shows which website the cookie belongs to.
4. Path: The specific path on the domain where the cookie is used. This can restrict where the cookie is sent.
5. Expiry: The expiration date of the cookie. It shows when the cookie will be automatically deleted from your device. If there's no expiry date, the cookie is a "session cookie," meaning it will be deleted when you close your browser.
6. Secure: Indicates whether the cookie is only sent over secure connections (HTTPS). This is important for protecting sensitive information.
7. HttpOnly: A flag that tells the browser not to allow JavaScript to access the cookie. This helps prevent certain types of security threats.

Example of a cookie table

‍

How to use a cookie table

• Review the cookies: By looking at the cookie table, you can understand what types of information a website is storing on your device.
• Manage your privacy: If you're concerned about privacy, you can use the cookie table to identify and delete cookies that you don't want.
• Understand expiry: The expiry column helps you see how long each cookie will remain on your device. Session cookies disappear when you close your browser, while others might stay longer.

Why is this important?

Knowing about cookies and how to read a cookie table can help you make informed decisions about your online privacy. You can decide which cookies you're comfortable with and which ones you'd prefer to block or delete.

Automating cookie table creation using Selenium

In addition to understanding cookies and cookie tables, you can automate the process of creating a cookie table for any website using Selenium in Java. This section provides a step-by-step guide to automate the retrieval of cookies from a website and export them into an Excel file using Selenium WebDriver and Apache POI.

Prerequisites

• Java Development Kit (JDK): Ensure you have JDK installed.
• Maven: Ensure Maven is installed for dependency management.
• Selenium WebDriver: Add Selenium dependencies to your project.
• Apache POI library: Add Apache POI dependencies to your project for Excel file handling.
• Web Browser Driver: Ensure you have the appropriate WebDriver for your browser (e.g., chromedriver for Chrome).

Setup

1. Add Selenium and Apache POI dependencies in Maven project (pom.xml):

<dependencies>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-java</artifactId>
        <version>4.0.0</version> <!-- Use the latest version -->
    </dependency>
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>5.2.3</version> <!-- Use the latest version -->
    </dependency>
</dependencies>

Code example

The following Java program automates the process of retrieving cookies from a website and writing them to an Excel file:

1. Create properties file

The properties file is used to store and manage the cookie token values or other related configurations. Here, we have created a config.properties file. 

Here's how and why:

Website.url = https://www.unicef.org/
Main.domain = unicef.org

Cookies.strictly_necessary = session,OptanonConsent,Session
Cookies.performance = ga,gid,gat,_gcl_au,nmstat
Cookies.functional = language,pvCount,userEngCount
Cookies.advertising = __pdst,_fbp

• This makes it easier to update the tokens if they change, without needing to modify the codebase. 
• Keeping tokens in a single, central file reduces the risk of inconsistencies or errors.

2. WebDriver setup

A WebDriver instance is created, and the browser is launched. The website URL can be replaced with your target URL.

WebDriver driver = new ChromeDriver();

// Navigate to the website
driver.get("https://www.unicef.org/");

3. Cookie categorization

The cookies are fetched from the browser, and categorized into different sets (strictlyNecessaryCookies, performanceCookies, etc.). The categorization logic here is just an example; you can implement your own.

// Fetch all cookies
Set<Cookie> allCookies = driver.manage().getCookies();

// Sets to store categorized cookies
Set<Cookie> strictlyNecessaryCookies = new HashSet<>();
Set<Cookie> performanceCookies = new HashSet<>();
Set<Cookie> functionalCookies = new HashSet<>();
Set<Cookie> advertisingCookies = new HashSet<>();
Set<Cookie> thirdPartyCookies = new HashSet<>();

// Categorize cookies based on predefined rules
for (Cookie cookie : allCookies) {
   if (isStrictlyNecessary(cookie)) {
	strictlyNecessaryCookies.add(cookie);
    } else if (isPerformance(cookie)) {
	performanceCookies.add(cookie);
    } else if (isFunctional(cookie)) {
	functionalCookies.add(cookie);
    } else if (isAdvertise(cookie)) {
	advertisingCookies.add(cookie);
     } else if (isThirdParty(cookie)) {
	  thirdPartyCookies.add(cookie);
   }
}

• Token matching: Each category classification method (isStrictlyNecessary, isPerformance, etc.) checks if the cookie name contains any tokens specified in the config.properties file.

// Example method to classify strictly necessary cookies
private static boolean isStrictlyNecessary(Cookie cookie) {
   // Example logic: Cookies related to session management, CSRF tokens, etc.
   String necessaryTokens = config.getProperty("cookies.strictly_necessary");
   return containsAny(cookie, necessaryTokens);
}

4. Workbook creation

• A new Excel workbook is created using Apache POI, and separate sheets are added for each category of cookies.
• This method takes in the filename and the sets of cookies for each category.
• The workbook is then saved to the specified file.
• The updateCookieSheet method adds headers, and populates the rows with cookie data.
• The workbook is saved to a file named CookiesTable.xlsx.
• The workbook is closed in a finally block to release resources.

private static void writeCookiesToExcel(String fileName, Set<Cookie> 
strictlyNecessaryCookies, Set<Cookie> performanceCookies, Set<Cookie> 
functionalCookies, Set<Cookie> advertisingCookies, Set<Cookie> thirdPartyCookies) 
throws IOException {

	XSSFWorkbook workbook = new XSSFWorkbook();

	try {
	  // Create sheets for each cookie category
	  XSSFSheet strictlyNecessarySheet = workbook.createSheet("Strictly Necessary Cookies");
	  XSSFSheet performanceCookiesSheet = workbook.createSheet("Performance Cookies");
	  XSSFSheet functionalCookiesSheet = workbook.createSheet("Functional Cookies");
	  XSSFSheet advertisingCookiesSheet = workbook.createSheet("Advertising Cookies");
	  XSSFSheet thirdPartyCookiesSheet = workbook.createSheet("Third-Party Cookies");

	  updateCookieSheet(workbook, strictlyNecessarySheet, strictlyNecessaryCookies);
	  updateCookieSheet(workbook, performanceCookiesSheet, performanceCookies);
	  updateCookieSheet(workbook, functionalCookiesSheet, functionalCookies);
        updateCookieSheet(workbook, advertisingCookiesSheet, advertisingCookies);
        updateCookieSheet(workbook, thirdPartyCookiesSheet, thirdPartyCookies);

	// Write the workbook to a file
	try (FileOutputStream fileOut = new FileOutputStream(fileName))     {
		workbook.write(fileOut);
	}

	System.out.println("Cookies have been written to the Excel file successfully!");
	} catch (IOException e) {
		System.out.println("Failed to write cookies to the Excel file.");
		throw e;
	} finally {
		workbook.close();
	}
}

Output

• Excel file: The output is an Excel file named CookiesTable.xlsx with separate sheets for each cookie category that lists all the cookies retrieved from the specified website. Each sheet includes columns for the cookie name, value, domain, path, expiry, secure flag, and HttpOnly flag.

The code snippet is available on github.

Benefits

• Automated process: This script automates the tedious process of manually collecting cookie information.
• Export to Excel: The cookie data is exported to an Excel file, making it easy to review, share, and store.
• Useful for testing and analysis: This approach is valuable for QA engineers, developers, and analysts who need to inspect cookies as part of web testing or security assessments.

Conclusion

Understanding cookies and cookie tables empowers you to take control of your online experience. It helps you see what data websites are storing and how it's being used, which is crucial for maintaining your privacy and security online. By automating the creation of a cookie table with Selenium and Apache POI, you can streamline the process of capturing and analyzing cookie data from websites.